1

I've noticed that when standardizing or centering the regressor(s), the p value for the intercept becomes smaller. Does this always happen? If so, why?

I know that $$ var(\hat{\beta}) = \sigma^2(X^TX)^{-1} $$

and that the t-stat is

$$ t_i = \frac{\hat{\beta}_i}{SE(\hat{\beta}_i)} $$

I can't tell from this general equation, nor the ones for simple linear regression found in https://en.wikipedia.org/wiki/Simple_linear_regression why the $t_i$ will increase and thus result in a reduction in the p-value.

24n8
  • 847
  • 3
  • 13
  • https://stats.stackexchange.com/questions/4125/regression-proof-that-the-point-of-averages-x-y-lies-on-the-estimated-regressi offers a strong hint. – whuber May 04 '21 at 14:27

1 Answers1

1

When you center the predictors, the only coefficient that changes is the intercept. See this answer.

Particularly of notice, is the fact that

\begin{matrix} \hat{\beta_L} = \left[\matrix{n^{-1}\mathbf {1}_n^T \\ (X^{*T}CX^*)^{-1}X^{*T}C}\right]y & \hat{\beta_X} = \left[\matrix{ n^{-1}\mathbf {1}_n^T+n^{-1} (\mathbf {1}_n^TX^*(X^{*T}CX^*)^{-1}X^{*T})C \\ (X^{*T}CX^*)^{-1}X^{*T}C }\right]y \end{matrix}

It's not particularly obvious that the p-value will diminish, it can actually increase. But, for non-trivial transformation of the independent variables, it's almost guaranteed to change.

Firebug
  • 15,262
  • 5
  • 60
  • 127
  • I'll check out your derivation in the linked post in a little bit, but I just want to follow up on "it can actually increase." When can it increase? – 24n8 May 04 '21 at 14:36
  • 1
    Suppose that the estimated intercept for the centered model is exactly zero (there exist values of the means and variances of x and y for which this will be true). Then the centered p-value is 1, regardless of the standard deviation ... – Ben Bolker May 04 '21 at 15:20