2

Why do we use a fixed design analysis of regression coefficients, even for observational data, where the design is not fixed?

For instance: $Var[\hat \beta]=(X'X)^{-1}\sigma^2$ is conditional on $X$. Since $X$ is random in observational studies, this is an under estimate of the true $Var[\hat \beta]$.

Edit: As pointed out by @christoph-hanck, $(X'X)^{-1}$ cannot be, by definition, systematically smaller than $\mathbb{E}(X'X)^{-1}$. Question remains: why do we use fixed design standard errors, instead of random design standard errors?

JohnRos
  • 5,336
  • 26
  • 56
  • What is the alternative? You do not have the random design matrix, so you can't estimate unconditional variance – Aksakal Aug 02 '18 at 13:30
  • 1
    The issue isn't with how the data collected so much as it concerns *how the results will be used.* Exactly what use do you have in mind for a regression analysis where you are concerned about the distribution of the regressors? – whuber Aug 02 '18 at 14:07
  • @whuber: I want to do inference on the $\beta$'s. – JohnRos Aug 02 '18 at 14:10
  • @Aksakal: we know from [Stein's paradox](https://en.wikipedia.org/wiki/Stein%27s_example) that when estimating many parameters at once (such as $Var[X]$), we can do improve accuracy by regularizing. – JohnRos Aug 02 '18 at 14:11
  • what does regularization have to do with this? it's a wholly different subject – Aksakal Aug 02 '18 at 14:13
  • @Aksakal: not really. If $(X'X)^{-1}$ is an estimator of $\mathbb{E}(X'X)^{-1}$, then I suspect that for most loss functions $(X'X)^{-1}$ is inadmissible and can benefit from some regularization. – JohnRos Aug 02 '18 at 14:22
  • Why do you bring up loss functions here? They're irrelevant to your question, unless you mean something different from what you asked. – Aksakal Aug 02 '18 at 14:25
  • In this model the estimation of the betas, because they concern the *conditional* distribution of the response, is independent of the distribution of the explanatory variables. Your question currently states you want to make inferences about the *variance* of their *estimates,* which is a different issue--and is not one commonly considered in regression settings. – whuber Aug 02 '18 at 14:32
  • @whuber: that is exactly my point. What is it so common to do conditional inference, whereas in observational studies, the design is random? – JohnRos Aug 02 '18 at 14:34
  • My point is that your question ultimately concerns the reasons why people do observational studies--and that likely depends on the field of interest. Most of the literature I have seen that uses regression in observational studies is concerned with estimating the conditional distributions rather than the full distributions. Other techniques are used to estimate full distributions, ranging from correlation analysis to PCA and beyond. – whuber Aug 02 '18 at 15:16
  • @whuber: I can only assume that people do observational studies because a designed experiment is impossible. I am asking why they do an observational study, and analyze it like a designed experiment. – JohnRos Aug 02 '18 at 15:43
  • See https://stats.stackexchange.com/questions/144826/what-is-the-difference-between-conditioning-on-regressors-vs-treating-them-as-f/192746#192746 – kjetil b halvorsen Jul 07 '19 at 10:04
  • Maybe we should write more clear: $Var[\hat \beta|X]=(X'X)^{-1}\sigma^2$ – user158565 Jul 07 '19 at 19:33

1 Answers1

0

By the law of total variance, we may write $$ Var(\widehat{\beta})=E[Var(\widehat{\beta}|X)]+Var[E(\widehat{\beta}|X)] $$ As OLS is (under suitable assumptions) unbiased, $E(\widehat{\beta}|X)=\beta$, so that $Var[E(\widehat{\beta}|X)]=Var[\beta]=0$, implying $$ Var(\widehat{\beta})=E[Var(\widehat{\beta}|X)] $$ so that I am not sure in which sense $Var(\widehat{\beta}|X)$ would systematically underestimate $Var(\widehat{\beta})$.

Christoph Hanck
  • 25,948
  • 3
  • 57
  • 106
  • Are you asking why $(X'X)^{-1}-\mathbb{E}[(X'X)^{-1}]$ is negative definite matrix? I suspect it can be shown using [Anderson's Lemma](https://en.wikipedia.org/wiki/Anderson%27s_theorem). – JohnRos Aug 02 '18 at 11:53
  • I suppose so. For concreteness, let us say we have iidness with a single standard normal regressor, so that $(X'X)^{-1}$ is inverse chi square with $E[(X'X)^{-1}]=1/(n-2)$ and we may have $(X'X)^{-1}>1/(n-2)$ or $(X'X)^{-1}<1/(n-2)$, no?` – Christoph Hanck Aug 02 '18 at 12:00
  • Oops. You are right that a R.V. cannot always be larger then its expectation (by definition) :-) That still does not answer the question: why do we use fixed design standard errors, even for random designs? – JohnRos Aug 02 '18 at 12:04
  • I think my answer points into that direction: because the fixed design variances are unbiased estimators of the unconditional variance of the OLS estimator. Couldn't you ask the same question of the OLS estimator itself: we may use it for fixed designs just as for random designs - it is conditionally and, by the law of iterated expectations, also unconditionally unbiased under random designs as well. – Christoph Hanck Aug 02 '18 at 12:10
  • This is not to say, of course, that other s.e.s might not also be useful - but how would "random design standard errors" look like? – Christoph Hanck Aug 02 '18 at 12:11