10

I don't understand why the probability of getting a type I error when performing a hypothesis test, isn't affected. Increasing $n$ $\Rightarrow$ decreases standard deviation $\Rightarrow$ make the normal distribution spike more at the true $µ$, and the area for the critical boundary should be decreased, but why isn't that the case?

(Cross posted on Math Stack Exchange.)

Community
  • 1
Stats
  • 143
  • 1
  • 1
  • 7
  • You choose $\alpha$, so in principle it can do what you like as sample size changes... and really, if you're minimizing the total cost of making the two types of error, it ought to go down as $n$ gets large. It makes no sense for people to keep using $\alpha=0.05$ (or whatever) while $\beta$ drops to ever more vanishingly small numbers when they get gigantic sample sizes. – Glen_b Dec 29 '14 at 14:06
  • 1
    I feel like I am missing some common point which you other guys has already understood. As far as I understand from the reponses is my theory correct, but the probability is kept eventhougt that isn't the case.. ??? – Stats Dec 29 '14 at 14:22
  • Limiting distribution of test statistic is unaffected by the sample size, I see no reason why one should decrease $\alpha$. Choice of $\alpha$ can be arbitrary. One can choose $\alpha=0.1$ for $n=10^{1000}$. Heart of the problem in frequentist statistics is whether the coverage probability of the level $1-\alpha$ confidence set is close to $1-\alpha$, for any given $\alpha$. – Khashaa Dec 29 '14 at 14:25
  • 4
    @Khashaa If you don't consider the type II error rate while choosing $\alpha$, you'll pay more (in terms of making more errors) than you need to. At sufficiently large sample sizes, power at some given effect size I care about will go arbitrarily close to 1 (0.99999...) -- at a much smaller sample size than we have (i.e. type II error will be as close to 0 as we like before we get to the current $n$). In that case, we can still attain that near-0 type 2 error at the larger sample size with fewer type I errors. I didn't say one *couldn't* choose $\alpha=0.1$, I was saying it's a bad idea ...ctd – Glen_b Dec 29 '14 at 15:12
  • (ctd) ... Unless you insist on making far more type I errors than necessary, I suppose. – Glen_b Dec 29 '14 at 15:16
  • @xtzx In your scenario, how are you deciding which cases are rejected? You seem to imagine a fixed rejection rule as you change $n$. But that's not how hypothesis tests work - you don't have a rule like "reject when the difference in means is 15" that applies to every sample size. – Glen_b Dec 29 '14 at 15:23
  • @Glen_b i don't really see how that is relevant... For me it seems intuitive that the type I error would decrease for increasing n,as the sampling distribution would tend spike more at the true µ, and samples which are far above or below will occur fewer times. the hypothesis will be rejected if the samples lies en critical region, which will occur fewer times as of the increasement of n. – Stats Dec 29 '14 at 15:28
  • 2
    @Glen_b I agree with you. One shouldn't choose only one $\alpha$. I am not very fond of the idea of "choosing $\alpha$". It carries a strange connotation as if $\alpha$ is some parameter inherent in the model. I'd be more interested in $1-\alpha$ level confidence intervals for range of $\alpha$ values. – Khashaa Dec 29 '14 at 15:35

3 Answers3

16

This is a question that is not asked often enough. In frequentist statistics we tend to fix $\alpha$ by convention. Then as $n\rightarrow\infty$ the type II error $\rightarrow 0$ (i.e., power $\rightarrow 1$) even though we also have the luxury for large $n$ of not allowing so many false positives had we chosen differently. The result of this convention is that when $n$ is "large", one can detect trivial differences, and when there are many hypotheses there is a multiplicity problem. By contrast, the likelihood school of inference tends to deal with the total of type I and type II errors, and lets type I error $\rightarrow 0$ as $n \rightarrow\infty$. This solves many of the problems of the frequentist paradigm. Ironically, the frequentist performance characteristics of the likelihood method are also quite good.

See for example http://people.musc.edu/~elg26/SCT2011/SCT2011.Blume.pdf and http://onlinelibrary.wiley.com/doi/10.1002/sim.1216/abstract .

Frank Harrell
  • 74,029
  • 5
  • 148
  • 322
  • 2
    These are great insights but could you please elaborate your answer a bit further by providing some references? (+1 anyway) – usεr11852 Dec 29 '14 at 13:14
  • 1
    I am not quite sure i understand the answer.. Is that the case or not, i am looking at it in inference manner.. – Stats Dec 29 '14 at 13:37
  • 2
    Yes $\alpha$ is traditionally kept constant as $n\rightarrow\infty$ but that is only by convention. – Frank Harrell Dec 29 '14 at 13:44
  • Ok.. that confuses me... So even though it seems "Logic" thing to say that probability of type I error decreases as n->$\infty$, it isn't the case, because it is kept ?? – Stats Dec 29 '14 at 13:51
  • 1
    Please re-read my comments. In traditional frequentist thinking the type I error probability does **not** decrease as $n$ increases. – Frank Harrell Dec 29 '14 at 18:44
7

It seems that you're missing the main point that Type I error rate is also your criterion for cutoff. If your criterion for cutoff is not changing then alpha is not changing.

The $p$-value is the conditional probability of observing an effect as large or larger than the one you found if the null is true. If you select a cutoff $p$-value of 0.05 for deciding that the null is not true then that 0.05 probability that it was true turns into your Type I error.

As an aside, this highlights why you cannot take the same test and set a cutoff for $\beta$. $\beta$ can only exist if the null was not true whereas the test value calculated assumes it is.

Frank Harrell's point is excellent that it depends on your philosophy. Nevertheless, even under frequentist statistics you can choose a lower criterion in advance and thereby change the rate of Type I error.

John
  • 21,167
  • 9
  • 48
  • 84
  • I was looking for something like this.. http://snag.gy/K8nQd.jpg – Stats Dec 29 '14 at 19:48
  • That highlighted passage does seem to contradict what has been said before, i.e. that α remains fixed to whatever you've set it to, and does not decrease with increasing sample size?! – z8080 Oct 12 '16 at 11:20
  • As I say to people who don't believe this, run a simulation. See the question marked as duplicate. – John Oct 12 '16 at 14:35
5

If you're using standard hypothesis testing, then you are setting the confidence level $\alpha$ then comparing the test p-value to it. In this case the sample size will not impact the probability of type I error because your confidence level $\alpha$ is the probability of type I error, pretty much by defintition. In other words, you set the probability of Type I error by choosing the confidence level.

The probability of type I error is only impacted by your choice of the confidence level and nothing else.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • I know that you predetermine what $\alpha$ should be. My question was more if changing the n would have an impact, which my textbook just confirmed it has, which also makes sense. $alpha$/level of significance changes as i change sample size. can't say how much though.. – Stats Dec 29 '14 at 21:14
  • @xtzx, did you look at the link I gave? It explains it all. – Aksakal Dec 29 '14 at 21:15
  • but the fact it changes the std. dev given by the link snag.gy/K8nQd.jpg, which also change the border line for the acceptance region, which will also affect $\alpha$ – Stats Dec 29 '14 at 21:25
  • 1
    @xtzx, nothing can change $\alpha$. You set it, only you can change it. – Aksakal Dec 29 '14 at 21:26
  • 2
    "..you are setting the confidence level $\alpha$.." I was always taught to use "significance level" for the $\alpha$ in a hypothesis test (e.g. 0.05), and use "confidence level" for confidence intervals (e.g. 95%). Having a quick look around the web suggests that's pretty much the universal terminology. – Silverfish Dec 30 '14 at 00:16
  • @Silverfish, correct – Aksakal Dec 30 '14 at 00:45