1

There is a question here on Stackexchange- Does the order of explanatory variables matter when calculating their regression coefficients?

Although I understand the details here, how can we formally prove that the order of variables doesn't change the coefficient estimates? I was trying to use partitioned matrices, and just showing that by changing the order of partitions and the respective coefficient estimates matrix, I get the same result. However, it doesn't look like a formal proof. Can anyone help?

Name LeftBlank
  • 35
  • 4
  • 2
    No proof is needed when you adopt the geometric point of view: the solution is the projection onto a subspace. This description makes no reference to the order of variables. If you want to be formal about it, this argument comes down to a linear algebraic result which is so trivial you may have a hard time finding it in any textbook: whenever $E=(e_1,e_2,\ldots, e_n)$ is a basis of a vector space, then so is $E^\prime=(e_{\sigma(1)}, e_{\sigma(2)}, \ldots, e_{\sigma(n)})$ for any permutation $\sigma$. – whuber Jan 16 '18 at 16:46

1 Answers1

2

Let $P$ be a permutation matrix such that $PP^T = P^TP = I$. It can be shown $PA = AP$ for any symmetric matrix $A$.

Then the permuted linear estimator $Y = P \mathbf{X}\beta $ has method of moments estimator $\hat{\beta}_P =\left( \mathbf{X}^T P^TP \mathbf{X} \right)^{-1} P\mathbf{X}^TY = P \left( \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^TY$ which is just the permutation of the unpermuted least squares estimator.

AdamO
  • 52,330
  • 5
  • 104
  • 209
  • 1
    I think it's even more clear using the optimization problem LR solves. – Matthew Drury Jan 16 '18 at 17:41
  • There may be more than one "answer" to the problem. Mine can just be the elementary proof. – AdamO Jan 16 '18 at 17:45
  • The disadvantage of this elementary proof is that it applies only to full-rank problems. Another disadvantage is that it proves something other than what was asked: you have changed the order of *observations*, not the order of variables! (Moreover, you neglected to permute the responses $Y$ in parallel, which is necessary to obtain the correct result.) Finally, your assertion that permutation matrices commute with all symmetric matrices is false. There are simple counterexamples with $3\times 3$ matrices. (Bear in mind the symmetric group $\mathfrak{S}_n$ is not commutative for $n\ge 3$.) – whuber Jan 16 '18 at 19:09