10

Based solely on the temporal pattern of mouse clicks (a list of click times $[t_1,t_2,t_3,\ldots]$), is it possible to predict the computer user's activity?

For example out of: working vs spending time on Facebook vs watching photos vs playing a computer game.

If they are any more fine-grained predictions (e.g. playing StarCraft vs Counter Strike vs SimCity) then I am interested as well.

While (arguably) one can hear is someone is playing (due to fast and bursty clicks) or watching photos (equally spaced clicks), I am interested if there are any more objective findings (publications, research on blogs, etc) on that matter.

EDIT:

I am equally interested in the keyboard clicks (without distinguishing which key is being hit) or the combined approach (mouse + keyboard).

Piotr Migdal
  • 5,586
  • 2
  • 26
  • 70

1 Answers1

4

Great question that I wish I had the time to investigate myself. I am confident that it is tractable. Do you have any data?

Your signal is a multidimensional ($n$D for $n$ buttons) binary times series; each bit indicating whether or not the button is depressed. You could also incorporate the position of the cursor into the feature vector as a 2D trajectory. Presumably, you have training data for each activity. So this means you have a classification problem.

You can reduce the dimensionality by approximating and efficiently encoding the trajectory (references on request), and taking the first-difference of the mouse click frequency (i.e., if the frequency of clicks is not changing, store zero). I would also estimate the distribution of the inter-arrival time of the clicks to see if you can classify from it.

For a jumping point into the literature see Activity recognition using eye-gaze movements and traditional interactions. You should find more leads in the "ubiquitous/pervasive computing", and "human–computer interaction" communities.

To obtain data I suggest generating it yourself using a keylogger. I suggest asking for help on a forum related to computer security or hacking. Most of them log the keyboard but there might be something for the mouse too. Failing that, you could write your own software.

Emre
  • 2,564
  • 15
  • 22
  • Good suggestions but isn't this more appropriately a comment rather than an answer since it doesn't answer the OPs question unless you think the link you gave is a partial answer? – Michael R. Chernick May 18 '12 at 23:05
  • The answer isn't the link; it's the paragraphs preceding it: I explained how to approach the problem and suggested the feature vector. If it doesn't provide good separation we can debate the finer points of the classification algorithm. – Emre May 18 '12 at 23:26
  • I guess that is a partial answer. The presumption is that the higher the frequency of mouse clicks the greater the activity. but what defines activity? Paasive things like reading can be considered activity and scrolling is an activity. Neither one involves mouse clicks. The real question is once we have a clear definition of what constitutes activity, we need an independent way to measure it and then see if it correlates with mouse clicking frequency. It probably does because sleeping in fron of the computer or staring at it give periods of inactivity and no mouse clicking. – Michael R. Chernick May 18 '12 at 23:46
  • On the other hand typing fast is a heavy form of activity but does not involve mise clicking at all. – Michael R. Chernick May 18 '12 at 23:47
  • @Emre Thanks for you suggestions, esp. the article. However, I am interested if it does actually work (I believe - yes). I don't have date, so links to such would be desirable as well. – Piotr Migdal May 19 '12 at 10:13