1

In linear regression when considering the f test for slopes be it MLR or SLR, we only consider the upper tail probability for the p value. Here's a video for reference: link. Why is that?

EDIT: as explained by the top answer below, it seems to be a likelihood test. My orignial understanding of the F test was similar to a two sided t test in terms of regression coefficients due to the alternative hypothesis. So then, why should we not view it as such?

Christoph Hanck
  • 25,948
  • 3
  • 57
  • 106
user2793618
  • 169
  • 5
  • In light of my comments about being interested in differences *at least as large*, why would you still think this would be a two sided test? – Demetri Pananos Nov 19 '20 at 06:57
  • I understand it from that point of view, but I just can't get over the alternative hypothesis for whatever reason. And differences are the differences between coefficients? – user2793618 Nov 19 '20 at 07:01

2 Answers2

3

If you look at the F statistic (e.g. Proof that F-statistic follows F-distribution) you may observe that it is a quadratic form - so negative as well as positive differences get squared to produce large positive values.

Hence, large values of the test statistic provide evidence that the restrictions imposed by the null are false. Small values of the test statistic, in turn, imply that the estimates are very close to the values implied by the null, hence no evidence at all against the null.

Now, a large value of the test statistic which is constructed as a square may arise both when the true coefficient is less that the value specified in the null and when the true coefficient is larger than the value specified in the null. Hence, the (default version of the) F test indeed tests the two-sided null that the underlying coefficients (or more precisely, at least one of them) are different from zero.

By the "default version", I mean the one that tests that the slope coefficients other than the coefficient on the constant are zero, $H_0:\boldsymbol{\beta}=\boldsymbol{0}$. But the general idea does not depend on that in that you can of course test all sorts of multiple linear restrictions with an F-test, and the large values of the F test statistic indicate significant departures from the null hypothesis in either direction.

Due to squaring, you however no longer know in which direction the departure arose (nor, in the F-test, from which coefficient(s)). Much like when you seek the number that, upon squaring, gave you the value 4, you do not know if it is -2 or 2.

Christoph Hanck
  • 25,948
  • 3
  • 57
  • 106
2

In linear regression, the F test is essentially a likelihood ratio test between your model and a model with a single parameter (the intercept). More parameters mean a larger likelihood, but the question is "is the change in the likelihood larger than I would expect if the true value of the additional parameters was 0?"

In essence, we do an upper tailed test because the question we are asking about is about differences being at least as large if not larger as we expect.

Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94
  • But isn't the alternative hypothesis the coefficients in question not being 0? And I'm not super familiar with likelihood ratio tests, could you explain that a bit more? – user2793618 Nov 19 '20 at 06:43
  • Ah it's a GOF test, why would it be considered a GOF test? – user2793618 Nov 19 '20 at 06:43
  • 1
    The F test can also be viewed as testing the marginal variance to the conditional variance. A smaller conditional variance (i.e. Larger F stat) might indicate goodness of fit because in some sense the parameters have "explained" some variance. As you add more parameters, the conditional variance must decrease, ergo one tail test. I don't think its a particularly good GOF test. It tests if there is any signal in the variables you've included in your model. – Demetri Pananos Nov 19 '20 at 06:46
  • Why would we wish for the conditional variance to decrease? And how should I get around getting stuck on the alternative hypothesis? Why should I not view it akin to a two sided t-test? – user2793618 Nov 19 '20 at 06:49
  • Seems like you've got more questions than your original post alluded to. I suggest you edit the post to be clearer and have more detail. – Demetri Pananos Nov 19 '20 at 06:50
  • Yeah I'll do that now. – user2793618 Nov 19 '20 at 06:50
  • Just updated the OP. – user2793618 Nov 19 '20 at 06:52