Regression on population values?

Question

In the theory of test equating (see, e.g., Test Equating, Scaling, and Linking by Kolen and Brennan (2014)), suppose we have a group of examinees whom we randomly assign one of two exam forms, say form X and form Y.

Let $X$ be a random variable representing scores of form X, and similarly for $Y$. Letting $\sigma(X)$ denote the standard deviation of a random variable $X$, linear equating assumes that $$\dfrac{X - \mathbb{E}[X]}{\sigma(X)} = \dfrac{Y - \mathbb{E}[Y]}{\sigma(Y)}$$ with probability one. To interpret the above, suppose we have a particular score $X = x$ from form X; we would use the equation above to yield an equivalent score for form Y.

Solving for $Y$ in the above equation yields $$Y = \dfrac{\sigma(Y)}{\sigma(X)}\cdot X + \mathbb{E}[Y] - \dfrac{\sigma(Y)}{\sigma(X)}\cdot \mathbb{E}[X]\text{.}\tag{1}$$ This can be interpreted as a linear equation of $X$, with slope $\dfrac{\sigma(Y)}{\sigma(X)}$ and intercept $\mathbb{E}[Y] - \dfrac{\sigma(Y)}{\sigma(X)}\cdot \mathbb{E}[X]$.

The question I have is on $(1)$ above - in Kolen and Brennan, toward the end of section 3.3, they state:

The equation for linear equating... is deceptively like a linear regression equation. The difference is that, for linear regression, the $\sigma(Y)/\sigma(X)$ terms are multiplied by the correlation between $X$ and $Y$.

I am very familiar with linear regression and that $\hat{\beta}_1 = r \cdot \dfrac{S_y}{S_x}$ when working with sample data points, where $S_y$ and $S_x$ denote the sample standard deviations.

But how does this make sense in a population context - to the point where we're working with population quantities such as $\sigma(X)$ and $\sigma(Y)$? In particular, what function are we minimizing so as to obtain that $\rho_{X, Y} \cdot \sigma(Y)/\sigma(X)$ (note: these are population quantities, not the sample ones) would be the least-squares regression slope between $X$ and $Y$?

Answer 1

I offer this post as one possible way to unify the concepts of (ordinary) linear regression for a random sample and for population. I will take you through a sequence of concepts, from broadest to narrowest, finally arriving at what you're looking for.

Corresponding with two conceptions of regression, which differ on whether the explanatory variable $X$ is considered a random variable, are two conceptions of "population."

1. The population is a function $\mathcal P$ from a set $X\subset \mathbb R$ into a set of random variables. When $x\in X,$ let $Y_x$ designate the random variable associated with $x.$

2. The population is a bivariate random variable $(X,Y).$

"Regression" most broadly means the process of associating with each $x$ the expectation of the random variable attached to $x.$

1. The "regression of $Y$ on $X$" in the first case is the function $x\to E[\mathcal{P}(x)].$ It exists when all these expectations exist.

2. The "regression of $Y$ on $X$" in the second case is the conditional expectation $E[Y\mid X].$ The "function" $x\to E[Y\mid X=x]$ is not actually well-defined, but any two such functions can disagree only on a set of zero probability."

"Linear regression" means finding a linear function to approximate the regression in the least squares sense. That is, in some sense the average squared difference between the $Y$ values and what the linear function "predicts" is minimized. In both cases we may express this function in the form

$$f(x;\theta) = \beta_0 + \beta_1 x$$

where $\theta = (\beta_0, \beta_1)$ are the parameters. The two situations differ subtly in what "mean square difference" might mean.

1. In the first case, the expected squared difference between $f(x;\theta)$ and the random variable $Y_x$ is (of course) $E\left[\left(Y_x - f(x;\theta)\right)^2\right].$ This isn't enough, though: evidently we need some way to average the expected squared differences over the set $X.$ That implies there is some measure $\lambda$ defined on $X$ so that we may define the mean squared difference as $$\operatorname{MSE}(\theta)=\int_X E\left[\left(Y_x - f(x;\theta)\right)^2\right]\,\mathrm{d}\lambda(x).$$ For minimization purposes, any positive multiple of $\lambda$ will yield the same linear regression.

2. In the second case, we already have a probability measure (given by the marginal distribution of $X$). Thus, $$\operatorname{MSE}(\theta)=E\left[E\left[\left(Y - f(X;\theta)\right)^2\mid X\right]\right] = E\left[\left(Y - f(X;\theta)\right)^2\right].$$

Both cases can be framed identically in the language of Euclidean vector spaces. The vector space in question has at most three dimensions: it is generated by $Y,$ $X,$ and the constant function $1,$ which I will write as $\mathbf{1}.$ The Euclidean norm in the second case is given by

$$||V||^2 = E\left[V^2\right]$$

where $V$ is any linear combination of $Y,$ $X,$ and $\mathbf 1.$

It determines the inner product

$$\langle U,V\rangle = \frac{1}{4}\left(||U+V||^2 - ||U-V||^2\right) = E[UV].$$

For convenience in the following calculations, let's rescale this norm to make $||\mathbf{1}||^2 = 1.$

The least squares objective is the squared distance between $Y$ and the linear combination $\beta_0\mathbf{1} + \beta_1 X.$ The shortest distances occur when the residual $Y - (\beta_0 \mathbf{1} + \beta_1 X)$ is orthogonal to the subspace generated by $\mathbf{1}$ and $X.$ Orthogonality can be checked by establishing that the residual is orthogonal (separately) to each of those vectors. This gives a pair of simultaneous linear equations, the Normal Equations

$$\left\{\begin{aligned} 0 &= \langle Y - (\beta_0\mathbf{1} + \beta_1 X), \mathbf{1}\rangle &= \langle Y, \mathbf{1}\rangle - \beta_0 - \beta_1 \langle X, \mathbf{1}\rangle \\ 0 &= \langle Y - (\beta_0\mathbf{1} + \beta_1 X), X\rangle &= \langle Y, X\rangle - \beta_0\langle \mathbf{1}, X\rangle - \beta_1 \langle X, X\rangle \end{aligned} \right.$$

Writing $\bar X = \langle X, \mathbf{1}\rangle = \langle \mathbf{1}, X \rangle$ and similarly for $\bar Y,$ let

$$V(X) = ||X||^2 - \left(\bar X\right)^2;\quad V(Y) = ||Y||^2 - \left(\bar Y\right)^2;\quad \operatorname{Cov}(X,Y) = \langle Y, X\rangle - \left(\bar Y\right)\left(\bar X\right).$$

Assuming $V(X)\ne 0,$ the unique solution is

$$\hat \beta_1 = \frac{\operatorname{Cov}(Y,X)}{V(X)}$$

from which $\hat \beta_0$ is readily computed.

Notice that in the second formulation, the correlation coefficient is given by

$$r = \frac{\operatorname{Cov}(Y,X)}{\sigma(Y)\,\sigma(X)}$$

where $\sigma^2(X) = V(X) = S_X^2$ and $\sigma^2(Y)=V(Y)=S_Y^2.$ In this notation the solution given in the question is

$$\hat\beta_1 = r\frac{S_Y}{S_X} = r\frac{\sigma(Y)}{\sigma(X)} = \frac{\operatorname{Cov}(Y,X)}{\sigma(Y)\,\sigma(X)}\frac{\sigma(Y)}{\sigma(X)} = \frac{\operatorname{Cov}(Y,X)}{V(X)},$$

algebraically equivalent to the solution obtained above.

---

Any (nonempty) finite dataset $((X_1,Y_1), (X_2,Y_2),\ldots, (X_n,Y_n))$ determines its empirical (bivariate) distribution. It is a discrete distribution in which the probability of any ordered pair $(x,y)$ is $1/n$ times the number of observations equal to $(x,y).$ The foregoing integrals reduce to finite sums, where for any $V = (v_1,v_2,\ldots, v_n)$ that is a linear combination of $Y,$, $X,$ and $\mathbf{1}=(1,1,\ldots),$

$$||V||^2 = E\left[V^2\right] = \sum_{i=1}^n \frac{1}{n} V_i^2 = \frac{1}{n}\sum_{i=1}^n V_i^2.$$

You will recognize this as the mean square. It follows that $\bar X = (X_1+X_2+\cdots+X_n)/n$ is the usual mean and $V(X)$ is the population variance (the normalization factor is $1/n$ rather than $1/(n-1)$). However, since the normalization factors cancel in the fraction $\hat\beta_1 = \operatorname{Cov}(Y,X)/V(X),$ the estimate is identical to the Ordinary Least Squares estimate.

Answer 2

This answer is likely wrong, and I would appreciate any corrections.

This is the only way I can reason it: we assume $(X, Y)$ is uniformly distributed over $\{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\} \subset \mathbb{R}^2$. Then, $X$ is a random variable whose values are uniformly distributed in $\{x_1, \dots, x_n\}$ and $Y$ is a random variable taking on values from a uniform distribution taking on values in $\{y_1, \dots, y_n\}$.

Minimize the residual sum of squares $$R(\beta_0, \beta_1) = \sum_{i=1}^{n}[y_i-(\beta_0+\beta_1x_i)]^2\text{.}$$ This would result in the standard linear regression coefficient equations \begin{align*} \hat{\beta}_0&=\sum_{i=1}^{n}y_i \cdot \dfrac{1}{n} - \hat{\beta}_1 \cdot \sum_{i=1}^{n}x_i \cdot \dfrac{1}{n} \\ &= \mathbb{E}[Y] - \hat\beta_1 \cdot \mathbb{E}[X] \\ \hat{\beta}_1 &= \dfrac{\sum_{i=1}^{n}(x_i - \mathbb{E}[X])(y_i - \mathbb{E}[Y])}{\sum_{i=1}^{n}(x_i - \mathbb{E}[X])^2}\tag{1} \\ &= \dfrac{\frac{1}{n}\sum_{i=1}^{n}(x_i - \mathbb{E}[X])(y_i - \mathbb{E}[Y])}{\frac{1}{n}\sum_{i=1}^{n}(x_i - \mathbb{E}[X])^2} \\ &= \dfrac{\mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]}{\mathbb{E}[(X - \mathbb{E}[X])^2]} \\ &= \dfrac{\text{Cov}(X, Y)}{\text{Var}(X)} \\ &= \dfrac{\text{Cov}(X, Y)}{\sigma(X)} \cdot \dfrac{\sigma(Y)}{\sigma(X)\sigma(Y)} \\ &= \dfrac{\text{Cov}(X, Y)}{\sigma(X)\sigma(Y)} \cdot \dfrac{\sigma(Y)}{\sigma(X)} \\ &= \rho_{X, Y} \cdot \dfrac{\sigma(Y)}{\sigma(X)} \end{align*} as desired.

The line $(1)$ is true because $\sum_{i=1}^{n}x_i \cdot \dfrac{1}{n} = \mathbb{E}[X]$ by the uniform assumption.