2

I need advice about how to carry out an ANOVA. I studied some theory of ANOVA, but apparently it is not enough.

Basically I collected around 300 reaction times for 12 subjects in my experiment. For each subject I had 2 conditions. The first has 3 levels, the second has 5 levels. Each subject went through all the possible level combinations, so I have 50 reaction times for each combination (15 different combinations), for each subject.

I want to prove the statistical significance of the different reaction time with different condition. What is not clear to me is the following point: Should I use ALL the dataset, or only the MEAN RT for each subject / condition (that is, 15 data points for each subject) ? If I use the whole dataset, I have about 10,000 data points. If I do so, the degrees of freedom for the Error is 10,557, which confuses me.

In the reaction time papers that I have around they report an F(x,y) where this x and y are usually really small, not more than 100, but their dataset is usually extremely big, more or less like mine. This make me think that they don't actually use all the reaction times, but only the mean reaction times (I suppose that, during the computation of the ANOVA, the mean of the means is then calculated).

But does this make sense, statistically?

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Vaaal
  • 525
  • 3
  • 14
  • I doubt you should do ANOVA at all; you probably want some form of multilevel model. The other papers might also be doing this, which could account for the df in the F test. Your data are not independent. – Peter Flom Nov 21 '13 at 00:01
  • Unfortunately this is not the case, it seems that the paper actually uses ANOVA. For example: "A two factor within-subject ANOVA was performed. There was a clear effect of condition (F(2,18)=106.97 p<0.0001)" http://csjarchive.cogsci.rpi.edu/proceedings/2009/papers/524/paper524.pdf pag 3, under Results. Their design is quite similar to mine. I use repeated measure as well (which is the same as within subject, I suppose). As said in the paper, they use 20 subjects, with 240 trials for each subject. This make their dataset quite big, but the reported degree of freedom is quite small. – Vaaal Nov 21 '13 at 00:08

2 Answers2

2

It is quite common to aggregate reaction times, but there are downsides to this procedure (some of which can be quite substantial). Like @Peter says, by aggregating across trials you ignore the effect of learning, which might be of substantive interest. Three other things to keep in mind when dealing with repeated-measures reaction time data:

1) Reaction times often have positive skew, and can benefit from an inverse transformation (i.e., 1/RT). The nice thing with this transformation is that the units remain interpretable, as items/second rather than seconds/item.

2) Because of this skew, the mean might not be a good metric, because it is not a robust measure of central tendency. If an individual gets distracted during the experiment for one trial, this might potentially influence results.

3) There might be systematic variation found in the items. For example, simple math problems might take longer the more digits that are involved, regardless of how good you are at solving math problems.

This paper offers a good intro of how to analyze repeated-measures RT data using linear mixed-effects models. These models can also incorporate crossed random-effects to account for both by-participant and by-item variability simultaneously (more info here).

Of course, both skewness and systematic item variability might not exist in your sample. I'd recommend graphing the reaction times, seeing how an inverse transformation looks, and looking at the distributions of each item to get a better idea of what model to specify.

dmartin
  • 3,010
  • 3
  • 22
  • 27
1

Given your comments, yes, the other paper seems to be using the mean value.

This isn't wrong, but doesn't let you look at change over time. There's a nice writeup of this sort of analysis near the beginning of Hedeker & Gibbons

Peter Flom
  • 94,055
  • 35
  • 143
  • 276