Generalised least squares: from regression coefficients to correlation coefficients?

Question

For least squares with one predictor:

$y = \beta x + \epsilon$

If $x$ and $y$ are standardised prior to fitting (i.e. $\sim N(0,1)$), then:

• $\beta$ is the same as the Pearson correlation coefficient, $r$.
• $\beta$ is the same in the reflected regression: $x = \beta y + \epsilon$

For generalised least squares (GLS), does the same apply? I.e. if I standardise my data, can I obtained correlation coefficients directly from the regression coefficients?

From experimenting with data, the reflected GLS leads to different $\beta$ coefficients and also I'm not sure that I'm believing that the regression coefficients fit with my expected values for correlation. I know people quote GLS correlation coefficients, so I am wondering how they arrive at them and hence what they really mean?

Answer 1

The answer is yes, the linear regression coefficients are the correlations of the predictors with the response, but only if you use the correct coordinate system.

To see what I mean, recall that if $x_1, x_2, \ldots, x_n$ and $y$ are centered and standardized, then the correlation between each $x_i$ and $y$ is just the dot product $x_i^t y$. Also, the least squares solution to linear regression is

$$ \beta = (X^t X)^{-1} X^t y $$

If it so happens that $X^{t} X = I$ (the identity matrix) then

$$ \beta = X^t y $$

and we recover the correlation vector. It is often attractive to recast a regression problem in terms of predictors $\tilde{x}_i$ that satisfy $\tilde{X}^t \tilde{X} = I$ by finding appropriate linear combinations of the original predictors that make this relation true (or equivalently, a linear change of coordinates); these new predictors are called the principal components.

So overall, the answer to your question is yes, but only when the predictors are themselves uncorrelated. Otherwise, the expression

$$X^t X \beta = X^t y$$

shows that the betas must be mixed together with the correllations between the predictors themselves to recover the predictor-response correlations.

As a side note, this also explains why the result is always true for one variable linear regression. Once the predictor vector $x$ is standardized, then:

$$ x_0^t x = \sum_i x_{i} = 0 $$

where $x_0$ is the intercept vector of all ones. So the (two column) data matrix $X$ automatically satisfies $X^t X = I$, and the result follows.