1

Suppose you add a feature $x$ to an OLS model. The AIC goes down by 100 but the new feature is not statistically significant. The expanded model should be preferred overall but is the estimated coefficient on $x$ any good? According to statistical theory if it's not significant you can't reject the possibility that it's $0$.

badmax
  • 1,659
  • 7
  • 19

1 Answers1

7

According to statistical theory if it's not significant it's likely 0.

That is absolutely not what the theory says. First of all, effects of any variable are never really 0. They may be imperceptibly small, perhaps too small for instrumentation to detect, but never actually 0.

Second of all, the theory when properly interpreted really says that when the p value is greater than 0.05 (let's call it $p$), then there is a $p$ probability that we observe effects at least as large if not larger assuming the effect was truly 0 and all of our assumptions about the data generating process associated with the test, possibly including but not limited to: Normality, independence, heterogeneity of variance, asymptotics, etc. That is very different from what you've said.

Third of all, including or excluding a variable has little to do with significance as I explain here.

EDIT: I see you've edited your post to clarify and instead say " According to statistical theory if it's not significant you can't reject the possibility that it's 0."

And I'm still going to challenge this and say "what if my type one error rate was different from 0.05"?

The point is, collapsing inference to a single number (namely the p value) is THE original sin of statistics. There is so much more you can lean on to make this decision, and much of it will depend on what you're trying to do with that model. Now, I know you're not looking for a philosophical discussion (neither am I), and I imagine you're instead looking for pragmatic instruction on what to do in these cases. I don't think giving a set of "if/then" type rules for inference is particularly effective. So I'll ask a question in response: What do you intend to do with this model?

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94
  • I would like to estimate the effect size associated with this new variable. A statement like "by increasing the value of $x$ by 1 unit, $y$ goes up by $n$ units'. In regression $n$ would be the estimated coefficient $\beta$, but is still still valid under the conditions I describe? – badmax Aug 05 '20 at 23:27
  • @badmax absolutely. You can’t reject the null in the typical sense, but hypothesis testing has no effect on estimation. You’re free to make the estimates you mention. I would also construct a confidence interval and report that. – Demetri Pananos Aug 05 '20 at 23:49
  • 1
    I do wonder what we should make of a point estimate of $5$ but a confidence interval of $(-5, 15)$, though, especially if a value like $5$ would have practical significance. – Dave Aug 06 '20 at 01:49
  • @Dave that's exactly what I'm getting at. What can you say about a variable that adds a lot to model power but the confidence interval is straddling zero? – badmax Aug 06 '20 at 03:00
  • 1
    @Dave Frank Harrell and company have an interesting thread on this [here](https://discourse.datamethods.org/t/language-for-communicating-frequentist-results-about-treatment-effects/934). In your example, if the interval covered practically significant values then that would mean that practically significant values are consistent with the data. Additionally, a salient argument against assuming the effect is null is the classic "absence of evidence is not the absence of evidence", but this may have its own problems as has been argued [here](https://stats.stackexchange.com/a/476442/111259). – Demetri Pananos Aug 06 '20 at 03:19
  • 1
    "...when the p value is greater than 0.05 (let's call it p), then there is a p probability that we observe effects ..." Not to nitpick, but isn't this statement true no matter what value p has? – nope Aug 06 '20 at 04:49
  • @nope yep. I’m partially pointing out the absurdity of defining a cut off point – Demetri Pananos Aug 06 '20 at 11:46