10

I have been reading the description of ridge regression in Applied Linear Statistical Models, 5th Ed chapter 11. The ridge regression is done on body fat data available here.

The textbook matches the output in SAS, where the back transformed coefficients are given in the fitted model as:
$$ Y=-7.3978+0.5553X_1+0.3681X_2-0.1917X_3 $$

This is shown from SAS as:

proc reg data = ch7tab1a outest = temp outstb noprint;
  model y = x1-x3 / ridge = 0.02;
run;
quit;
proc print data = temp;
  where _ridge_ = 0.02 and y = -1;
  var y intercept x1 x2 x3;
run;
Obs     Y    Intercept       X1         X2         X3

 2     -1     -7.40343    0.55535    0.36814    -0.19163
 3     -1      0.00000    0.54633    0.37740    -0.13687

But R gives very different coefficients:

data <- read.table("http://www.cst.cmich.edu/users/lee1c/spss/V16_materials/DataSets_v16/BodyFat-TxtFormat.txt", 
                   sep=" ", header=FALSE)
data <- data[,c(1,3,5,7)]
colnames(data)<-c("x1","x2","x3","y")
ridge<-lm.ridge(y ~ ., data, lambda=0.02)   
ridge$coef
coef(ridge)

>   ridge$coef
       x1        x2        x3 
10.126984 -4.682273 -3.527010 
>   coef(ridge)
                   x1         x2         x3 
42.2181995  2.0683914 -0.9177207 -0.9921824 
>

Can anyone help me understand why?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
B_Miner
  • 7,560
  • 20
  • 81
  • 144
  • I have used the same data set and analyzed it in SPSS. The results are similar to SAS. However if we can standardized the data as in the book, the first value at c=0 is similar in R and SPSS (running a multiple linear regression). However the results are quite different when we move further as indicated above. I too found no clue of the difference between outputs for same data. – asad Jul 09 '14 at 23:01

2 Answers2

6

Though ridge regression looks at first like simple algorithm the devil is in the details. Apparently original variables are scaled, and parameter $\lambda$ is not the parameter you would think it is given the original description. From what I gathered reading the reference given in R help page of lm.ridge there is no one agreed way of doing ridge regression. So the difference in results can only be explained by different algorithms used by R and SAS. Hopefully someone more knowledgeable can give more detailed answer.

You can see what kind of algorithm is applied in R by looking at the source of lm.ridge. Just type lm.ridge in the R prompt.

mpiktas
  • 33,140
  • 5
  • 82
  • 138
  • I found this link helpful for deciphering how lm.ridge is calculating the coefficients: http://www.mail-archive.com/r-help@r-project.org/msg81115.html But still puzzled how different the results are from the text I referenced and SAS, given that each is supposedly back to the original scale. – B_Miner Jan 31 '11 at 13:13
  • @user2040, check that the same data is used by SAS and R. If it is the same, then the only conclusion is that the algorithms are different. What SAS help page is saying? – mpiktas Jan 31 '11 at 13:22
  • @user2040, I've replicated [SAS ridge regression](http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_reg_sect046.htm) on R with your data. So we know for sure that the algorithms are different. – mpiktas Jan 31 '11 at 13:59
  • 1
    You just beat me to that! :) I was looking at the SAS help page you referenced. I compared the RMSE (in-sample data only, did not validate with a CV or bootstrap yet) and the R result was superior.So, do you think ridge regression is best suited for prediction and not interpretting the coefficients (since the results can be so different by algorithm)? I know already that regular linear model theory (CI's, contrasts etc.) are out for ridge regression (bootstrapping gets closer but still can be misleading due to bias). – B_Miner Jan 31 '11 at 16:06
0

Using lm.ridge also produces a scaling vector (try head(model) to see all the output). To get the predicted values in R that you see in SAS take the coefficients and divide by the scalar vector.

Thomas
  • 1