4

What exactly is the error that is made in the following judgement?

"I cannot increase my sample size, so I'll make up for the low number of subjects by asking the subjects to perform more trials, i.e. by increasing the total number of observations / data points" ?

Does an explanation based on degrees of freedom and the repeated measures vs between-subjects nature of the data collection explain this error of judgement?

z8080
  • 1,598
  • 1
  • 19
  • 38
  • 2
    How odd. This strategy is not *necessarily* an 'error of judgement'. It rather depends on variability in trial performance. – conjugateprior Jul 21 '15 at 13:29
  • I think the repeated measurments can increase your power, since people can change over time. people at different time may stand for different people. ha, this logic seems weird. – Deep North Jul 21 '15 at 13:56
  • 2
    @DeepNorth the same people certainly can *not* stand for different people... That is exactly what this question is about - within- and between- person measurements are different things that should not be confused. – Tim Jul 21 '15 at 14:21
  • Tim: no it is not homework it is just me wondering – z8080 Jul 22 '15 at 15:13

3 Answers3

8

Simple counter-example would be probably enough for you to figure the answer yourself. Say, that you have to make a study on human height, however you do not have enough time to conduct a full-scale survey. In this case you decide to ask your roommate 1000 times about his height.

Tim
  • 108,699
  • 20
  • 212
  • 390
  • 3
    +1 Although this intuition won't always apply... Here's a counter-example to your counter-example: To learn about corridor coffee preferences, ask $n$ students in an $N$ person corridor how much they'd like a cup of coffee $k=1$ times. Increasing $n$ will help, but so will increasing $k$ by asking them multiple times in a day, because expressed coffee preferences are variable due to time of day, time since the last one, etc. – conjugateprior Jul 21 '15 at 13:28
  • 2
    Although it's already obvious in this case, you did learn about the within person variance in regards to height. – TrynnaDoStat Jul 21 '15 at 15:11
  • @TrynnaDoStat yes, but the "study" was not on the within person variation so it would lead to wrong conclusions. – Tim Jul 21 '15 at 20:17
  • Tim: I guess I should have given more background. The noise that I'm trying to reduce with the repeated measurements is not due to individual differences but with the noise of the perceptual system. Therefore, while in your example replacing the k=1000 observations from N=1000 subjects with the same number of observations from N=1 subject would not make sense, in my case doing something like this *might* make sense, although it still feels very dodgy – z8080 Jul 22 '15 at 15:28
6

Asking the same person to perform the same trial will increase your information about within-person variance but not between-person variance. If within-person variance is what you're interested in, then the logic makes sense. If it's not what you're interested in, then it doesn't.

TrynnaDoStat
  • 7,414
  • 3
  • 23
  • 39
  • 4
    It will increase within-person variance, but may also *increase* measurement precision in some cases, e.g. you construct a questionnaire that asks about *similar* issues (using *different* questions), so that averaged answers would provide a more reliable measure then when using a single question. This is often used in social sciences. – Tim Jul 21 '15 at 13:46
  • This is probably the answer I was looking for, thanks TrynnaDoStat – z8080 Jul 22 '15 at 15:29
2

Another way to look at this is to consider that all the measures you take reflect a systematic source of variance (which you are interested in) compounded by various sources of errors (i.e. variance you are not interested in).

Depending on your objectives and the relative magnitude of these sources of errors, different designs might make sense but you can't learn about or compensate one source of error (e.g. individual differences between participants) by increasing the number of observations in another facet of the design (e.g. repeated observations of the same participant over time).

Generalizability theory is a way to formalize this insight (and the source of the terminology I used in this answer).

Gala
  • 8,323
  • 2
  • 28
  • 42
  • Excellent explanation Gala. Would it also be correct to say that by collecting additional observations from new subjects as opposed to from the same subject, I am increasing the _external validity_ of the inference? – z8080 Jul 22 '15 at 15:35