prcomp() vs lm() results in R

Question

I have a simple matrix:

     [,1] [,2] [,3]
[1,]    1    2    3
[2,]    4    5    6
[3,]    7    8    9
[4,]   10   11   12

I have to calculate linear regression and orthogonal regression using lm() and prcomp() respectively. (for orthogonal see: here)

Assume that the first column is the the X and M the matrix I wrote before.

LINEAR REG.

mod1 <- lm(M[,1] ~ M[,2] + M[,3] + 0)

Its output is (coefficient):

Coefficients: M[, 2]  M[, 3]  
     2      -1

Ok, I have these coefficients.

Now for

ORTHOGONAL REG.

mod2 <- prcomp(~ M[,1] + M[,2] + M[,3])

Its output is:

             PC1        PC2        PC3
M[, 1] 0.5773503  0.0000000  0.8164966
M[, 2] 0.5773503 -0.7071068 -0.4082483
M[, 3] 0.5773503  0.7071068 -0.4082483

The question is: out to interpret prcomp() result instead of lm() result ? Using lm() the coefficients are using to predict the X values.

What about prcomp() ?

Thank you!

I also think you need to confirm you have quoted the correct coefficients for the `lm()`. I get `-1, 1, NA` for the data/model you show, which is correct given the linear dependencies in the example data. Don't forget that the intercept is also a coefficient, so your model actually has three coefficients. — Gavin Simpson, Jul 27 '11 at 10:56
prcomp **is not** doing orthogonal regression, at most can be used as an element of its implementation. Not to mention it is probably not worth it. — , Jul 27 '11 at 11:55
@ mbq did you see the list i posted? It talk about orthogonal regression using prcomp()....btw what is the function to calculate the orthogonal regression in R? — Dail, Jul 27 '11 at 12:00
You might want to start by reading the highly-rated thread on PCA at http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues. — whuber, Jul 27 '11 at 13:43
I admit this crazy concept of dismantling PCA into some OLS regression is widely present, however this does not mean you should expect PCA functions to do regression. About the second thing -- none; this is the same stuff as [here](http://stats.stackexchange.com/questions/13152/how-princomp-works/13167#13167). — , Jul 27 '11 at 14:34
the output of prcomp is loading on eigenvectors, meaning you first rotate your data around most variate axis and then these are the loading for your data, But lm gives you the loading for current axis without any rotation. — Areza, Jul 28 '11 at 14:06
@mbq - PCA using the first principal component is just as valid as regression method as OLS - they are both methods for fitting straight lines. OLS minimises the error in the $Y$ direction only $S(\beta_0,\beta_1,\dots,\beta_p)=\sum_i(Y_i-\hat{Y}_i)^2$, whereas the first principal component minimises the "perpendicular error" $S(\alpha_0,\alpha_1,\dots,\alpha_p)=\sum_i(Y_i-\hat{Y}_i)^2+\sum_j\sum_i(X_{ij}-\hat{X}_{ij})^2$. — probabilityislogic, Sep 23 '11 at 13:09
@probabilityislogic I say it is not a regression in a sense it does not make any regressive model, rather a coordinate system transformation. Whether it can be used as a part of some linear regression method is an orthogonal problem. — , Sep 23 '11 at 19:50

score 1 · Answer 1 · answered Sep 23 '11 at 17:23

Although you have used $M_2$ and $M_3$ in your example, you effectively have an intercept because $M_2+1=M_3$, and so $$M_{1}=M_{2}\beta_2+M_{3}\beta_3=M_{2}\beta_2+(M_{2}+1)\beta_3=M_{2}(\beta_2+\beta_3)+\beta_3$$ (This is confirmed in @Gavin's comment, as $\beta_3=-1$ and $\beta_2+\beta_3=1$.). So your coefficient for $M_3$ is the intercept for the model with only $M_2$, and your coefficient for $M_2$ is the negative intercept for the model with only $M_3$.

In terms of a general comparison, you are comparing two straight lines (or points on straight lines). For a lm() model, we have

$$y_{i}=\beta_0+{x}_{i1}\beta_{1}+\dots+x_{ip}\beta_{p}$$

This line will pass through the point $(\beta_0,0,0,\dots,0)$ and have slopes in each direction of $(1,\beta_1,\beta_2,\dots,\beta_p)$. We can equivalently state this as passing through the point $(\beta_0+\overline{x}_1\beta_{1}+\dots+\overline{x}_p\beta_{p},\overline{x}_1,\dots,\overline{x}_p)$

Now in order to put this into the principal components analysis framework. Note that here we are only looking at the first PC. You are given a vector in $p+1$ dimensional space, $(\alpha_{Y},\alpha_{1},\dots,\alpha_{p})$, which is the principal component. This component describes a straight line, which passes through the centroid of the data $(\overline{y},\overline{x}_1,\dots,\overline{x}_p)$ and has slopes in each direction of $(\alpha_Y,\alpha_1,\dots,\alpha_p)$. Because slopes only need be defined in proportion (just alters the size of the line), we can restate this as $(1,\frac{\alpha_1}{\alpha_Y},\dots,\frac{\alpha_p}{\alpha_Y})$ (for $\alpha_Y\neq 0$ of course). This means $\beta_j$ is the equivalent quantity to $\frac{\alpha_{j}}{\alpha_Y}$. This is how you can compare the results

I am not sure I understand how OLS/lm() produces a "line". It fits the equation $$y_{i}=\beta_0+{x}_{i1}\beta_{1}+\dots+x_{ip}\beta_{p},$$ as you wrote, but how is that a line? Isn't it a $p$-dimensional hyperplane? — amoeba, Feb 03 '15 at 15:13
In continuation of my previous comment: I posted an answer in the (arguably duplicate) thread that discusses PCA and TLS. See here: [How to perform orthogonal regression (total least squares) via PCA?](http://stats.stackexchange.com/a/136597/28666) I continue to believe that your answer, as it is formulated now, is wrong. In fact, in multiple TLS regression one needs to look at the *last* and not at the first PCA eigenvector, in order to obtain the beta coefficients. — amoeba, Feb 06 '15 at 22:32

prcomp() vs lm() results in R

1 Answers1