0

I have a question: Does something will be wrong, if me, in addition to comparing the control and experimental groups, I will make an analysis inside these groups? Someone pointed this out to me, noting the need for patches for multiple comparisions. I do not agree with this because you cannot talk about the same family of hypotheses when you test on different samples (one control gropup, second:experimental). I would ask forum users to answer and explain why I am wrong if I am not.

(edit): To clarify what I mean, example:

I have two variables, for example weight and height. First, I test hypotheses about differences in height and in weight between control and experimental group (but not a relationship between weigh and height on the whole sample!). Then I test the relationship between weight and height in the experimental group. Then the same in the control group.

Rafał
  • 73
  • 4
  • 1
    You need to correct the p-values because you are using the same data. First you test on the whole data, and then based on the results you test on some subset of it, but it is still the same data. – user2974951 Jan 09 '20 at 11:51
  • Hmmmm... OK, but I think this would only be the case if I made the same comparisons on the whole sample and then on the subsets: control and experimental group. I meant something else. The best way to explain what I meant it would be give you an example. I have two variables, for example weight and height. First, I test hypotheses about differences in height and weight between control and experimental group. Then I test the relationship between weight and height in the experimental group. Then the same in the control group. Which hypotheses are from the same family here? – Rafał Jan 09 '20 at 18:39
  • It is the same data, but completely different comparisions – Rafał Jan 09 '20 at 19:14
  • 1
    It doesn't make a difference. Taken to an extreme, one can devise thousands of comparisons and ultimately some of them will turn out to be "significant" for any given dataset. Indeed, for the kinds of comparisons you envision and these kinds of data, if you are conducting your tests with $1-\alpha$ confidence then you can expect a proportion $\alpha$ of them to be significant *even when there are no differences whatsoever between the groups.* Please visit https://stats.stackexchange.com/questions/88065/explain-the-xkcd-jelly-bean-comic-what-makes-it-funny before going any further. – whuber Jan 09 '20 at 19:22
  • On the picture from that side, the fact that those hypothesis are from the same family is obviously to me. So it doesn't help me at all. I will be glad, if someone answer me on the simple question: which pair of hypotheses I mentioned should be treated as a pair of hypotheses from the same family? – Rafał Jan 09 '20 at 19:58

0 Answers0