Frequentist perspective of regression coefficients and significance (coming from Bayesian background)?

Question

I come from a primarily Bayesian background when using performing statistical analysis. In the context of linear regression, I would look at the posterior distributions for each regression coefficient. If a given coefficient was distributed ~N(1,1), I might not be terribly convinced of the associated variable's influence on the dependent variable.

From a frequentist perspective, regression coefficients often have p-values assigned. And I'm not quite sure what to make of this coming from a bayesian, full posterior interpretation. Is the idea that there's only an x% chance that we would have observed this specific regression coefficient inferred by accident?

Answer 1

A p value is the probability of observing a test statistic as or more extreme than the researcher's own test statistic, assuming the null hypothesis, and an assumed distribution model are both true.

So when you see frequentist statistical software provide regression output that includes something like:

name	estimate	S.E.	t (or z)	p value
cons	3.0	0.015	200	<0.001
beta	0.5	0.18	2.78	0.003

The default null hypothesis for regression coefficients in most stats software I am familiar with is $\text{H}_{0}\text{: }\beta = 0$, with $\text{H}_{\text{A}}\text{: }\beta \ne 0$. You should read the $p = 0.003$ as $p = P(|T_{\text{df}}|\ge |t|)$ given $\text{H}_{0}$, or in plain language: the probability of observing a value of $\widehat{\beta}$ as or more extreme than 0.5 is 0.003, assuming the null hypothesis is true (and assuming Student's t distribution truly describes the distribution of your test statistic). If a priori you had an $\alpha =0.01$, then you would interpret that p value as evidence against the null hypothesis, and for the alternative hypothesis (i.e. you found evidence that $\beta \ne 0$).

Answer 2

In most regression software you can also request confidence intervals for the regression coefficients. The confidence interval is a set of plausible true values of the parameter, given the observed data. A 90% confidence interval contains all the hypotheses for the parameter under which the statistic is not significant at the two-sided 10% level. While this does not have the same interpretation as a 90% credible interval, it serves the same purpose. If the 90% confidence interval contains zero (two-sided p-value testing $H_0:\beta=0$ is greater than 0.10) you would not be convinced at the two-sided 10% level of the association between the covariate and the dependent variable. If the 90% confidence interval excludes zero (two-sided p-value testing $H_0:\beta=0$ is less than 0.10) this could be interpreted as evidence (not proof) of an association at the two-sided 10% level (or one-sided 5% level).

If you use something like Proc MCMC you are used to getting kernel density estimates for the posterior distribution of a parameter. You can analogously produce confidence curves using the point estimate and standard error. These curves show p-values and confidence intervals of all levels for hypotheses concerning the parameter.

The upper-tailed p-value for a standard issue Wald test as a function of the hypothesis would take the form

$$H(\beta,\boldsymbol{x})=1-\Phi\bigg(\frac{\hat{\beta}-\beta}{\hat{\text{se}}}\bigg).$$

Analogously, the lower-tailed p-value is $$H^{-}(\beta,\boldsymbol{x})=\Phi\bigg(\frac{\hat{\beta}-\beta}{\hat{\text{se}}}\bigg).$$

The corresponding confidence curve, $C(\beta,\boldsymbol{x})$, can then be defined as

$$C(\beta,\boldsymbol{x})=H(\beta,\boldsymbol{x})\text{, if $\beta < \hat{\beta}$ }$$ $$C(\beta,\boldsymbol{x})=H^{-}(\beta,\boldsymbol{x})\text{, if $\beta > \hat{\beta}$ }$$.

The confidence curve will look like a teepee or a pyramid on the parameter space depicting ex-post sampling probability of the experiment (frequentist confidence).

If, say, you have an estimated beta coefficient of 0.5 and an estimated standard error of 0.18, the figure below depicts the corresponding confidence curve for inference on the unknown fixed true $\beta$.

data norm;
do beta=-0.6 to 1.6 by 0.001;
beta_hat=0.5;
se=0.18;

C_lower=1-cdf('normal',(beta_hat-beta)/se,0,1); if beta gt beta_hat then C_lower=.;
C_upper=cdf('normal',(beta_hat-beta)/se,0,1); if beta lt beta_hat then C_upper=.;

output;
end;
run;

proc sql noprint;
select max(beta)
into: lower
from norm
where . lt C_lower le 0.05;

select min(beta)
into: upper
from norm
where . lt C_upper le 0.05;
quit;

data norm;
set norm;
lower=&lower.;
upper=&upper.;
yscatter=-0.035;
run;

ods escapechar="^";
ods graphics / height=3in width=6in border=no;
proc sgplot data=norm noautolegend;
series x=beta y=C_lower / lineattrs=(color=darkblue) name="cc" legendlabel="Confidence curve (one-sided p-value) with 90% confidence interval";
series x=beta y=C_upper / lineattrs=(color=darkblue);
scatter x=beta_hat y=yscatter / xerrorlower=lower xerrorupper=upper errorbarattrs=(color=darkblue) markerattrs=(color=darkblue);
refline 0 / axis=y;
yaxis max=0.6 label="p-value" min=0 offsetmin=0.1;
xaxis label="^{unicode beta}" min=-0.5 max=1.5;
keylegend "cc";
run;

Answer 3

These numbers are unreliable if your predictors are correlated.

Suppose for example that the value to be predicted is the sum of two hidden N(0,1) random variables $Y=Z_1+Z_2$ and $Z_2$ is unobserved, and you have observations of $X_1 = Z_1+N(0, 0.01), X_2 = Z_1+N(0, 0.01)$

Then the linear regression is degenerate. Any solution is equally good, in which the coefficients of $X_1, X_2$ sum to one. No magic with p-values can help you choose.

Perfect degeneracy like this is rare, but some level of correlation is common. Take home lesson: correlation is not causation.