4

I have three subjects in a qualitative study. I am comparing the features of their writing before and after treatment.

I have no idea about the normality of the distribution since they were selected purposefully. I have counted the number of everything (like the complex sentences, correct sentences, etc.) and now I want to compare each case in the subjects before and after treatment. When you look at the numbers, the difference is too high (like for example around 800 before the treatment but more than 2000 after). T-test results showed a p-value of 0.017 while wilcoxon showed 0.109. Hence t-test showed a significant difference while wilcoxon showed a nonsignificant difference.

Now my question is, due to the small sample size (only three) and not being sure of the normality of distribution, is t-test a correct procedure? I have also run chi square and it has shown a significant difference.

chl
  • 50,972
  • 18
  • 205
  • 364
Farhad Rahimi
  • 41
  • 1
  • 2
  • Is it the case that you have multiple dependent variables & you're checking all of them? You are likely to find something just by virtue of checking many things. OTOH, you will have pretty severe issues w/ statistical power (ie, the ability to achieve significance) in many cases. My hunch is that normality is the least of your issues. Drop any pretense of statistically testing your data; simply present them & assess them qualitatively. – gung - Reinstate Monica Oct 29 '12 at 02:31
  • Hi. Actually, I agree with you to some extent. But suppose that I had only one variable, i.e. the complexity of writing before and after the treatment operationalized by counting the t-units in the three students' essays before and after treatment, and, hence, coming to sth like 600 for students one before the treatment and 2100 for the same student after the treatment and more or less the same thing for the other two. What statistical procedure would you use? t-test, wilcoxon, or simply chi-square? Thanks – Farhad Rahimi Oct 29 '12 at 23:53
  • If you were going to do only one test, the question is whether you are willing to make the theoretical assumption of normality. If so, you can use the t-test. If not, then either the Wilcoxon or even the sign test. The Wilcoxon can actually be more powerful than the t-test when the data aren't normal, I believe, but you're typically not going to have much power. – gung - Reinstate Monica Oct 30 '12 at 02:52
  • Thanks. But you see, the problem is that although the difference between the two figers is too large (600 and 2100) and the quality of the students' writing show much improvement, the wilcoxon does not yield significant results, while those of t-test are significant. – Farhad Rahimi Oct 30 '12 at 05:12
  • I recognize that, but we don't pick which test to use by checking to see which one will give us the results we want. I would just drop any pretense to statistically testing your data and simply present them & analyze them qualitatively. Another possibility is to run Bayesian analyses starting from a couple of prior belief states that people might find reasonable, & show what you ought to believe now given the prior & what you found. – gung - Reinstate Monica Oct 30 '12 at 13:27

1 Answers1

2

(I believe I should summarize the comments in an answer, lest this question go 'officially' unanswered.)

It sounds like you have multiple dependent variables and you're checking all of them. This is likely to be a strategy that leads you into trouble. You will find something just by virtue of checking many things. On the other hand, with just three sets of paired data, you will have pretty severe issues with statistical power in general, and doubly so if you make any attempt to control for the possibility of inflated type I errors. My hunch is that normality is the least of your issues.

If you had only one variable to test, the question is whether you are willing to make the theoretical assumption of normality. (This is purely an assumption because you do not have enough data to get any reasonable information about the shape of the distribution.) If you can assume this, however, you can use the t-test; there is certainly no problem with having only 3 data points (see: Is there a minimum sample size required for the t-test to be valid?, for a recent discussion of this issue). If you are not comfortable making the assumption of normality, then either the Wilcoxon or even the sign test will be acceptable. The Wilcoxon can actually be more powerful than the t-test when the data aren't normal, I believe, but you're still not likely to have much power with just three points per test. I recognize that you have already checked and found significance with the t-test but not with the Wilcoxon. However, a pretty basic logical principle is that we can't pick which test to use by checking first to see which one will give us the results we want.

My suggestion would be to just drop any pretense to statistically testing your data and simply present them in total, and discuss them qualitatively. Another possibility that might be worthwhile, if you are sufficiently familiar with it, is to run Bayesian analyses starting from a couple of prior belief states that people might find reasonable. Then show what people with these beliefs ought to believe now given what you found. (To be honest, I think this is one of the prototypical situations when everyone, even diehard Frequentists, ought to find the Bayesian approach to be the most useful and practical option available.)

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650