4

I can not solve the problem from my homework.

We conducted two experiments. In the first, there were 400 patients, and in the second, 250. In these experiments, the effects of various drugs were evaluated. The average weight of people in the two groups was compared using the t-test. The test of normality for the first group gave a p-value below the significance threshold, and for the second above the significance threshold (but the histogram was bell-shaped). Variance in groups differed by more than 30%. Is it possible to say that the experiments were compared incorrectly?

There are answer answers to choose, only one of them is correct:

  1. everything is bad, the distribution is not normal, we need a Wilcoxon test
  2. everything is bad, the samples have different variance and sizes, you cannot use the t-test
  3. the samples are large enough so that everything is fine
  4. everything is fine, p-value of one of the groups is larger than the threshold, so the result of the first group could be random, then the distribution is still normal

Personally, I think that the correct answer is the second, because the condition for the applicability of the t-test is homogeneity of variance, and in these experiments it is very different. But I'm not sure that this is the right answer.

COOLSerdash
  • 25,317
  • 8
  • 73
  • 123
Anastasiia Zakarian
  • 41
  • 3
  • Although one of these answers may be correct, it is impossible to determine that from the information given, and it is conceivable that any one of the answers could be "the" correct one, depending on the details of the data and the experimental protocol. – whuber May 02 '19 at 14:16

2 Answers2

4

The question is very poorly constructed and contains some serious flaws. The context of the question may help. Looking at the 'rules' and 'guidelines' for two-sample t tests that have been given just previously to the question may help you figure out what the author means.

Major flaws are as follows:

  • "[T]he second [P-value is ] above the significance threshold (but the histogram was bell-shaped)." I agree with @statsandr (+1) that this seems self-contradictory.

  • "Variance in groups differed by more than 30%." The appropriate way to judge whether sample variances indicate the population variances may be unequal is to look at their ratio, not their difference.

  • Nothing is said about the difference in sample means and no clue is given how large a difference would be of practical importance. So, against what standard are we to judge an "incorrect" comparison?

Also, we don't know whether the two-sample t test under discussion is a 'pooled' or a 'Welch' test. A Welch test should take care of a difference in variances. The DF of a Welch test can't be below $\min[(n_1 - 1),(n_2 - 1)] = 249,$ so the t statistic must be nearly normal.

If a real-life situation, using a Welch t test, is described here, my guess is that everything is OK. But the exposition of the question is so foggy that my crystal ball doesn't say which answer its author expects.

BruceET
  • 47,896
  • 2
  • 28
  • 76
  • (+1) for mentionig the capability of the welch test (which I left out) and pointing out that "The question is very poorly constructed". Every option begins with a general statement ("everything is bad/ fine") what I find misleading, too. And I agree that it is not clear what option is correct –  May 01 '19 at 16:23
2

@1. The importance normality decsreases while N increases. See here or here. "With large enough sample sizes (> 30 or 40), the violation of the normality assumption should not cause major problems. This implies that we can use parametric procedures even when the data are not normally distributed" (from first source).

@2. This may be a problem since if sample sizes are unequal, unequal variances can influence the Type 1 error rate of the t-test by either increasing or decreasing the Type 1. Still, you may want run a Levene's test or to see how much the varianes differ.

@3: To my understanding big sample sizes don't guarantee that everything will be fine. Ok, we sad that with increasing sample size the normality assumption becomes less inportant in 1. I am not too sure about the same being true for unequal variance.

@ 4. I don't understand that sentence. And in fact, I find this sentence confusing, too "for the second above the significance threshold (but the histogram was bell-shaped)". Why does it say "but"? A significant test for normality would indicate data not being normal not the other way. At least this is the case for tests I know like the Shapiro-Wilk test where "The null-hypothesis of this test is that the population is normally distributed. Thus, on the one hand, if the p value is less than the chosen alpha level, then the null hypothesis is rejected and there is evidence that the data tested are not normally distributed." (source)

EDIT

Thanks to @Glen_b who pointed out that the quote under "@1" should be restrained because "while the t-test may end up having a nice normal-looking null distribution in many cases if n is large enough, its performance under the null isn't really what people care most about -- it's performance under the alternative -- and there it may not be so great, if you care about rejecting the null in the cases where the effect is not so easy to pick up." (quote from Glen_b)

  • 1
    (+1) for pointing out self-contradictory phrase and for useful links. – BruceET May 01 '19 at 15:38
  • re your "@1" ... I don't see that there's clear agreement at the source with the quoted part. Some of the other answers and comments to that question (and clear demonstrations in answers to other questions on site) should cause us to attach very strong doubt the the claim unless very carefully qualified. In particular (a) while the importance of normality to the significance level decreases with increasing sample size, it isn't true for *power* (as several people there take the trouble to point out), and (b) claims relating to any specific $n$ are easily shown to be false in general – Glen_b May 02 '19 at 04:26
  • @Glen_b: Thank you. The quote is from the first link. I changed that. And indeed, I made the mistake to only read the most upvoted answert where it is said that "The t-test is invalid for small samples from non-normal distributions, but it is valid for large samples from non-normal distributions.". But you're right that point 1 of my answer is too general if one considers the power problems. However, I don't understand what you mean with (b). For example, if the claim is that the standard error in a t test decreases when n increases why that would be shown wrong? –  May 02 '19 at 11:15
  • In relation to (b), I said "a specific $n$" -- that is, mention of specific samples sizes, as in your quote which says "> 30 or 40". That claim - that if n exceeds 40, violation of the normality assumption will not cause problems - is not true. By contrast, you're just now talking about "what happens as $n$ increases", which is not at all the same thing -- and in particular isn't claiming something is necessarily 'close enough' by some *particular* $n$. Very different kind of statement to the thing I am responding to. – Glen_b May 02 '19 at 13:02
  • @Glen_b: Now I understand what you mean with b). Thank you! –  May 02 '19 at 13:09
  • A wide variety of similar claims of that kind will be false. As a general rule of thumb, a claim that relates to distribution shape of some statistic like a mean at a specific sample size that does not carefully address some facts about the distribution (e.g. its third absolute moment divided by the cube of the standard deviation) will usually be wrong – Glen_b May 02 '19 at 15:05