0

Say, for instance, I'm estimating a model with about 5 million observations using linear regression or MLE. Given that the estimates are consistent, using the standard rule of rejecting the null on a 5% significance level will pretty much never make me entitled to reject the null (of the parameter being equal to zero), i.e. everything is significant.

I should probably use another significance level, which is smaller than 5%, and my question is if there is any literature on picking the right significance level, or how would you guys handle this problem? Should I simply stick with the 5% significance level?

Thanks.

Nick Stauner
  • 11,558
  • 5
  • 47
  • 105
user37008
  • 56
  • 5
  • You parameter is never exactly zero; why do you need to reject the obviously wrong? I'll give you a reference to Tukey on how to formulate good hypotheses later today. How do you estimate your model "using linear regression or MLE"? The rule of 5% is by no means a standard one - just a convenient one - can refer you to a good paper co-authored by Peter Hall but again later. – Hibernating Jan 08 '14 at 04:14
  • Thanks for the good answer. I'm not interested in the parameter being exactly equal to zero, this will happen with probability 0, but I'm interested whether it is indistinguishable from zero. I'm referring to the 5 % significance level as a standard rule, because in 99 % of the articles in economics/econometrics this is used without further discussion. Regarding my question, it is not of interest to me whether I do MLE or linear regression as this is just a question referring to standard inference, i.e. doing t-test, chi-square test or similar. – user37008 Jan 08 '14 at 12:51
  • I am afraid you need to ask another question. A quick search shows something like How to formulate a good null hypothesis has not been asked yet. This will allow me to deploy Tukey. Ask about history of that golden rule of stats (p-value > 0.05) and I will refer you to Hall. In many situations hypothesis testing is less than appropriate regardless of the size of your data. – Hibernating Jan 09 '14 at 00:57

1 Answers1

0

  1. The first (default) method to correct for the "multiple comparisons" problem is to use a Bonferroni adjustment to the p-value. The new $\alpha$ level is $\alpha^*=\frac{\alpha}{\#tests}$. Thus, divide the Type I error level of $\alpha=0.05$ by the number of tests you have.

  2. Use the Benjamini-Hochberg False Discovery Rate approach (1995), which is less conservative.

  • 1
    I assumed this question was about the increased likelihood that even a negligibly small or spurious effect could achieve significance in a large data problem, but you might be right to address the separate issue of [multiple comparisons](http://stats.stackexchange.com/q/28712/32036), depending on whether the OP wants to control for this kind of error inflation. See also [an issue with multiple comparisons corrections](http://stats.stackexchange.com/q/1458/32036). – Nick Stauner Jan 08 '14 at 04:40