0

I have run a DIF analysis of a questionnaire, to see how autistic men and women answer differently, with a total sample of 530 autistic men and 530 autistic women. I have a general hypothesis that the questionnaire is biased towards men. I have removed all problematic items (all 40 of them) to create a shorter (supposedly more fair) questionnaire.

I have carried out a t-test at the beginning (the difference between men and women's scores was very significant at p < 0.001). I have then carried out a t-test at the end, but on the same sample that I ran through the DIF analysis (there is still a significant difference in scores with p < 0.05).

Is there an issue with running a t-test on the same sample? Because DIF is exploratory, I feel like maybe I should should not be running a t-test on the very same sample that created the shorter questionnaire in the first place. I could remove maybe 50 men and 50 women from the original sample, and put them aside for after the DIF analysis, but this would make my sample size significantly smaller, and there are no previous studies of it so no way of knowing standard error/what an adequate sample size is.

I am starting to think that it's not an issue anyway, because if anything this is good evidence that there IS an actual difference in their scores.

Let me know what you think,

Thanks,

Thea

T August
  • 21
  • 3
  • Are differences roughly normal? Is it the same 1060 subjects throughout? If so, is it a problem that participants will recognize that some questions (all but 40) are the same on the second run? Are you planning to compare the early differences with the later ones? – BruceET Apr 01 '20 at 00:38
  • Hello, the differences slightly normal and it is the same 1060 participants. I won't be doing the questionnaire again, but rather using their original data (that went through the DIF analysis in the first place) – T August Apr 04 '20 at 13:44

0 Answers0