Weighted Least Squares Through The Origin

Question

For a regression model through the origin, with Var($e_i|x_i) = x_i^2σ^2$ . The corresponding regression model is $Y_i$ = $\beta$$X_i$ +$e_i$. How do I create a least squares model? I know I need to take the derivative of $\sum w_i(y_i-\hat{y_i})^2$. Is the weight $\frac{1}{(X_i)^2}$?

Answer 1

This is a linear regression model with heteroskedastic (and I presume non-autocorrelated) error terms, with the functional form of heteroskedasticity known. In such a case, things are pretty easy, because the structure of the covariance-matrix of the error term is known, and so we can implement Generalized Least Squares (not "Feasible" such).

What should the weights be? The purpose of the weights is to transform all the variables involved in the equation in such a way so as the transformed error term has constant variance. Denote this weight $w_i$ (to be determined). Then we are looking at

$$w_iy_i = \beta w_ix_i+w_ie_i \Rightarrow \tilde y_i= \beta \tilde x_i+\tilde e_i$$

We want

$$\text{Var}(\tilde e_i \mid \tilde x_i) = \sigma^2 \Rightarrow E[\tilde e_i^2 \mid \tilde x_i]=\sigma^2$$

$$\Rightarrow E[(w_ie_i)^2 \mid w_ix_i)=\sigma^2 \Rightarrow w_i^2E[e_i^2\mid w_ix_i] = \sigma^2 \Rightarrow w_i^2\cdot (x_i^2\sigma^2) = \sigma^2$$

The only way for this to hold is if we set $$w_i^2 = \frac 1{x_i^2} \Rightarrow w_i = \frac 1{|x_i|}$$

As provided in the post linked to by a comment, for the initial equation $y_i = \beta x_i+e_i$ we have

$$\hat{\beta}_{OLS}=\frac{\sum_{i=1}^N x_iy_i}{\sum_{i=1}^N x_i^2}$$

Then for our transformed model we have

$$\hat{\beta}_{GLS}=\frac{\sum_{i=1}^N \tilde x_i\tilde y_i}{\sum_{i=1}^N \tilde x_i^2} = \frac{\sum_{i=1}^N \frac{x_i}{|x_i|}\frac{y_i}{|x_i|}}{\sum_{i=1}^N \frac {x_i^2}{|x_i|^2}} = \frac 1N\sum_{i=1}^N \left(\frac{y_i}{x_i}\right)$$

Note that implicit in all the above is that the regressor does not take zero values (otherwise one could apply a correction, but we will then be faced with a possibly very large variance for the observation involved).

Using $y_i = \beta x_i+e_i$ we can arrive at $$\hat{\beta}_{GLS} = \beta + \frac 1N \sum_{i=1}^N \left(\frac{e_i}{x_i}\right)$$

which gives

$$\text{Var}(\hat \beta_{GLS} \mid \mathbf x) = \frac 1{N^2}\sum_{i=1}^N \left(\frac{\text{Var}(e_i \mid \mathbf x)}{x_i^2}\right) = \sigma^2/N$$

This should be anticipated, since

$$\tilde x_i = \frac {x_i}{|x_i|} \Rightarrow \tilde x_i^2 = 1$$

and, as a general result for a simple regression without a constant,

$$\text{Var}(\hat \beta_{GLS} \mid \mathbf x) = \frac{\sigma^2}{\sum_{i=1}^N \tilde x_i^2}$$

Moreover, given this estimator, we know that the expression

$$\frac 1{N-1}\sum_{i=1}^N \hat {\tilde e_i}^2,\;\; \hat {\tilde e_i} = \tilde y_i - \hat{\beta}_{GLS}\tilde x_i$$ is a meaningfull estimator of the unknown constant $\sigma^2$.