1

I have 5 different samples(P1,P2,P3,P4 and P5) with 4 replicates(a,b,c,d).I did paired t-tests with the null hypothesis that there is no significant difference when compared to the control group at p=0.05.

After the t-tests, I performed Holm-Bonferroni correction for one sample at a time(considering 4 replicates of each sample at a time) to adjust the p-values. This resulted in the table of values as shown in the figure. Do I reject the null hypothesis based on the adjusted p-values when p <0.05(or can say the difference is significant)?

enter image description here

shome
  • 115
  • 5
  • After adjustment only P5 is significant at $\alpha=0.05$. – user2974951 Feb 07 '20 at 10:40
  • @user2974951 Can you please give me suggestions about the added values(just edited the post),as well ? Also,if you can suggest any specific literature I should look into for better understanding ? – shome Feb 07 '20 at 11:08
  • 1
    @shome what is your hypothesis ? – peuhp Feb 07 '20 at 11:18
  • @peuhp I have added the hypothesis and background.Please have a look.Are there any more details from my end ? – shome Feb 07 '20 at 11:32
  • You have not one hypothesis. You tested many hypothesis and have to decide for each hypothesis if you reject it based on the p-value. – Roland Feb 07 '20 at 12:01

1 Answers1

1

Your setting looks like an explanatory analysis (as opposed to a confirmatory study, and this is not a problem in itself). I would suggest you not to focus too much on significance and null hypothesis rejection but to try to go back to your experimental setting. For that purpose, I would suggest :

  • first, look at the effect size associated to each comparison.
  • second, think about what their relative values tell you about your different experimental settings. Are they surprising ? waited for ? It there some particular explanation to it ?
  • Try to thing at a larger scale. Can you group evidences? Are all your tests really testing different things?

In your setting, having a few low p-values is something that may be indicative that what you are looking at may not be noise, which may be a good thing. Use p-values to discriminate results that may deserve interpretations and do not use them to conclude authoritative statement such as "H0 is false". IMHO, there is no real need to adjust them if you don't abuse their meaning.

peuhp
  • 4,622
  • 20
  • 38