4

A multiple linear regression model is considered. It is assumed that $$ Y_i = \beta_1x_{i1} + \beta_2x_{i2} + \beta_3x_{i3} + \epsilon_i$$ where $\epsilon$-s are independent and have the same normal distribution with zero expectation and unknown variance $\sigma^2$. 100 measurements are made, i.e $i = 1,2,..., 100.$ The explanatory variables take the following values: $x_{i1} = 2$ for $1 \leq i \leq 25$ and $0$ otherwise, $x_{i2} = \sqrt{2}$ for $26 \leq i \leq 75$ and $0$ otherwise, $x_{i3} = 2$ for $76 \leq i \leq 100$ and $0$ otherwise.

a) Let $\hat{\beta_1},\hat{\beta_2}, \hat{\beta_3}$ be least squares estimators of $\beta_1, \beta_2, \beta_3$. Prove that in the considered case $\hat{\beta_1},\hat{\beta_2}, \hat{\beta_3}$ are independent, and $$Var(\hat{\beta_1}) = Var(\hat{\beta_3}) = Var(\hat{\beta_3})$$ Do these properties hold in the general case? If not, give counterexamples.

b) Perform a test for $$H_0: \beta_1 + \beta_3 = 2\beta_2$$vs.$$H_1: \beta_1 + \beta_3 \neq 2\beta_2$$ The significance level is 0.05. The least squares estimates of $\beta_1, \beta_2$ and $\beta_3$ are $0.9812, 1.8851$ and $3.4406$, respectively. The unbiased estimate of the variance $\sigma^2$ is $3.27$.

For a) I know the OLS estimator for $\hat{\beta} = (X^TX)^{-1}X^Ty$, and $Var(\hat{\beta}) = \sigma^2 (X^TX)^{-1}$. But I don't know how to attain explicit expressions for each of the coefficients from this. Although it seems quite clear that the estimators are independent, for instance $P(\hat{\beta_3} = \beta_3, \hat{\beta_1} = 0, \hat{\beta_2} = 0) = P(\hat{\beta_3} = \beta_3)$ but I don't how to write a proper proof. I believe the estimators are generally dependent and have unequal variance, but can't come up with any particular examples.

For b) not sure what test-statistic to use (t or F) and how to set it up. Also don't know the standard errors of the coefficients

Community
  • 1
Pame
  • 157
  • 7

2 Answers2

4

You have already given the formula for the variance matrix for the coefficient estimator. The Gramian matrix of the design matrix for the regression ---which appears in that formula--- is:

$$\begin{aligned} \mathbf{x}^\text{T} \mathbf{x} &= \begin{bmatrix} \mathbf{x}_1 \cdot \mathbf{x}_1 & \mathbf{x}_1 \cdot \mathbf{x}_2 & \mathbf{x}_1 \cdot \mathbf{x}_3 \\ \mathbf{x}_2 \cdot \mathbf{x}_1 & \mathbf{x}_2 \cdot \mathbf{x}_2 & \mathbf{x}_2 \cdot \mathbf{x}_3 \\ \mathbf{x}_3 \cdot \mathbf{x}_1 & \mathbf{x}_3 \cdot \mathbf{x}_2 & \mathbf{x}_3 \cdot \mathbf{x}_3 \\ \end{bmatrix} \\[6pt] &= \begin{bmatrix} \sum_{i=1}^{25} 2 \times 2 & 0 & 0 \\ 0 & \sum_{i=26}^{75} \sqrt{2} \times \sqrt{2} & 0 \\ 0 & 0 & \sum_{i=76}^{100} 2 \times 2 \\ \end{bmatrix} \\[6pt] &= \begin{bmatrix} 25 \times 4 & 0 & 0 \\ 0 & 50 \times 2 & 0 \\ 0 & 0 & 25 \times 4 \\ \end{bmatrix} \\[6pt] &= 100 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}. \\[6pt] \end{aligned}$$

This is proportionate to the identity matrix, which means that the coefficient estimators are uncorrelated with equal variance. You therefore have:

$$\mathbb{V}(\hat{\boldsymbol{\beta}}) = \sigma^2 (\mathbf{x}^\text{T} \mathbf{x})^{-1} = \frac{\sigma^2}{100} \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{bmatrix}. \\[6pt] $$

You should be able to derive the coefficient estimates from the standard regression formula and then use these to formulate the hypothesis tests. Each hypothesis test is testing a linear combination of the coefficients, so you can use the rules for linear combinations of normal random variables to derive the standard errors of the test statistics.

Ben
  • 91,027
  • 3
  • 150
  • 376
  • Is it possible to prove that the coefficient estimators being uncorrelated implies they are independent in this case? Or would you have to use a different method to prove independence? – Pame Apr 17 '20 at 12:38
  • 1
    @Pame It’s a property of Multivariate Normal Distribution that if the components are pairwise uncorrelated then they are Independent. You can even verify this by putting $\rho = 0$ for a bivariate normal distribution and see that it reduces into a product of 2 univariate normal distribution. – Vishaal Sudarsan Apr 17 '20 at 13:48
  • Are the $\beta$ coefficients generally multivariate normal for multiple linear regression, or is it special for this case? I understand they are univariate normal due to the residuals being normal. – Pame Apr 24 '20 at 09:21
2

As You know that the OLS estimator is a linear function of $y$. $\hat{\beta} \sim N(\beta, \sigma^2(X’X)^{-1})$ since $\epsilon \sim N(0,\sigma^2)$.

1) All you need to show is that the matrix $(X’X)^{-1}$ is a scalar matrix. Just compute the inverse using appropriate entries for $x_1,x_2,x_3$

2) All you need to do is test the following Hypothesis :-

$H_0 : l’{\beta} = 0$ vs $H_1 : l’{\beta} \neq 0$

Where $l’$ is (1,-2,1).

Now use a $t$ test on $l’\hat{\beta}$ which follows a univariate normal distribution.

Vishaal Sudarsan
  • 520
  • 2
  • 9
  • For 2) with the t-statistic don´t you get $SE(l' \beta$) in the denominator which is $0$? – Pame Apr 12 '20 at 17:02
  • @Pame I haven’t done the calculations but the unbiased estimate of variance is given to be 3.27 and $Var(\hat{\beta_1} -2\hat{\beta_2} + \hat{\beta_3}) = 6*Var(\hat{\beta_1})$ since the variance is same for all \hat{\beta_i} and they’re independent as shown in 1). so your Standard error would be 0 only if the $Var(\hat{\beta})$ is 0. Which will happen only if $(X’X)^{-1}$ which is impossible. So to answer your question, the denominator won’t be 0 you must be going wrong somewhere. – Vishaal Sudarsan Apr 12 '20 at 18:39