1

I am planning an evaluation which will deploy pre- and post-tests for one group of students. I would like to perform a dependent t-test but may not be able to perform a repeated measures analysis since we may not be able to collect full names, maybe only ages from the students.

Is there any meaningful difference between a matched pairs (e.g., pairing by ages) versus repeated measures analysis? My preference would be to perform a repeated measures analysis if we could ask the students to enter their full names. If we can't collect names and have to do a matched pairs analysis using ages, would there be any disadvantage in doing so?

Thanks in advance for any feedback.

Lucas
  • 11
  • 4

1 Answers1

1

Your question is really unclear.

  1. How many measures (i.e., tests of students) do you intend to deploy? Two? More than two?
  2. Are the repeated measures able to be matched up for each student? That is to say, can you somehow collect tests in a way that you can tell that a pair (or group if the answer to item 1 above is more than two) of tests were completed by the same student? Can you anonymize them with an ID?
  3. What exactly do you mean by "pair by age?" In a typical student population, you should expect many students to be of the same age. What happens if you have 5 students who are all the same age? How do you "pair up" their repeated measures? Are you suggesting you take the means of each measure in time for each observed age, and do a repeated measures test on each age? This is obviously problematic.
  4. What is the hypothesis you wish to test or the inference you wish to make?

What I can say is that you cannot do a matched pairs or repeated measures test if you can only identify pairs/groups of tests by age. The assumption for such tests requires that you can identify which observations are repeated measures on the same experimental unit.

heropup
  • 5,006
  • 1
  • 16
  • 25
  • 1) We are planning to deploy one pre-test, one post-test immediately after the intervention, and a second post-test 2-months after the intervention. 2) I agree that assigning an ID to each student would work but this approach may complicate the evaluation (which we are really trying to avoid). Without the students' names, we would not be able to track which scores are for which students. 3) That is true. So I could not randomly pair the students with the same ages, even if many students have the same age? (e.g., randomly pair two 7 yr. olds, until all 7 yr. olds have been paired)? – Lucas Nov 25 '15 at 23:38
  • ... 4) In the previous phase of the evaluation, I was able to perform a t-test for the pre-test and post-test scores since I had the students names, ages, locations, etc. I would like to be able to perform a t-test or similar analysis again but this time I may not be able to match the pre-test to post-test scores to any one student. So I guess my questions is, what alternatives to a t-test for repeated measures are there to determine if any changes in mean test scores are statistically significantly different between pre- and post-tests? – Lucas Nov 25 '15 at 23:46
  • No. You can't randomly pair them up. Paired or repeated measures tests assume that the test statistic follows a distribution corresponding to paired or matched observations. Randomly pairing them will violate that assumption and will lead to an incorrect inference. Don't do it. – heropup Nov 25 '15 at 23:54
  • Just do a regular analysis of variance, treating each repeated measure as independent, because there is no way to associate them longitudinally. Because you have three measurements instead of two, you also can't use a t-test, whether paired or two-sample independent. – heropup Nov 25 '15 at 23:57
  • Thank you for those clarifications and suggestions. I will look into AoV. Could not I still perform several t-tests, for example one for any changes between the pre-test and immediate post-test, another between the pre-test and 2-month post-test, etc.? – Lucas Nov 26 '15 at 00:01
  • You could do pairwise comparisons if your hypothesis is only interested in comparing the pre-test and post-test contrasts, which it sounds like it is. The ANOVA will test the general hypothesis that the means among pre-, post-1, and post-2 are not all equal. But if you do pairwise comparisons, you must adjust for multiplicity. In your case, this should be accomplished via Dunnett's test, although the conservative Bonferroni adjustment is simpler. – heropup Nov 26 '15 at 00:06
  • I hope you had a good Thanksgiving @heropup. As a follow-up, if I could get the students' names or IDs to perform a t-test for repeated measures, is there any evidence to support performing a t-test over an ANOVA of the unpaired data? In other words, would it be worth it (from a statistical perspective) to obtain the students names so that I could perform a t-test on of the pre-/post-test scores or would it be just as valid to "settle" for an ANOVA of the independent measures? – Lucas Nov 27 '15 at 16:45
  • @Lucas The paired test will have more statistical power to detect differences in means for the same sample size. – heropup Nov 27 '15 at 18:23
  • Related to your (@heropup) suggestion about doing "a regular analysis of variance, treating each repeated measure [pre-test, post-test, etc.] as independent," why would it be valid to treat each measure as independent if the group of students will be the same (even though the measures can't be paired)? – Lucas Nov 27 '15 at 22:13