2

My professor asked the following problem to drill in the ideas of significance levels and $p$-values during lectures and it's really doing my head in. Suppose I want to test a particular hypothesis $H_1$. Is there a difference between the two hypothesis testing scenarios?

  1. I pose 99 additional hypotheses $\{H\}_{2\leq i\leq 100}$, collect data, fail to reject null hypothesis on $\{H\}_{1\leq i\leq 99}$  at a 0.05 significance level, then get a $p$-value of 0.01 for $H_{100}$.

  2. I collect data, fail to reject null hypothesis on $H_1$ at a 0.05 significance level, pose a new hypothesis $G$ calculated on the collected data, immediately obtain a $p$-value of 0.01.

What can I claim about $H_{100}$ and $G$ at a 0.05 significance level or a 0.001 significance level? I'm comfortable with my understanding of when to reject for a single test, but when it comes to combining multiple hypothesis tests together I lose track of what we can and cannot do. Could someone please provide some insight into this? Will be eternally grateful!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
user107224
  • 211
  • 1
  • 3
  • 1
    It depends strongly on whether you want to evaluate the evidence concerning H1 with a significance test or whether you want to decide whether to reject H1. See this for a little more essential background: https://stats.stackexchange.com/questions/16218/what-is-the-difference-between-testing-of-hypothesis-and-test-of-significance/16227#16227 – Michael Lew Nov 07 '20 at 20:27
  • If you are going to perform a Neyman-Pearsonian hypothesis test then you must set the critical value (i.e. the "significance level") in advance of performing the test. You really need read a little more about hypothesis tests and significance tests before attempting to sort out anything as complicated as your question. – Michael Lew Nov 08 '20 at 04:03

0 Answers0