3

Let's say I am predicting weight from height (cm) and age (years). Then I decided to convert height into meters and age into months. The interpretation of the coefficients out of the regression might be not that straightforward (I will need to think in meters and months) but I will still get my accurate regression, won't I?

I also thought that the by converting the age into months might be bad for the prediction, as people's (grown-ups) weight do not change drastically within the months, it might change drastically within the years. So, I will get high variance and I might overfit, is that correct?

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
Anni
  • 449
  • 2
  • 5
  • 12

3 Answers3

8

Nature is indifferent to units of measurement. Thus there will be no untoward effects of changing the units of your explanatory variables in a regression model.

If people's weight increases over years, then it will increase, on average, 1/12 as much over months. Thus, the estimated slope in the model using months will be precisely 1/12 the estimated slope in the model using years. Likewise, if people who are taller in centimeters tend to be heavier, then those same people, when measured in meters, will also tend to have the same association between their height and weight, but the estimated slope will be 100 times larger.

Changes of units will not affect the goodness of fit of your model, its out of sample predictive accuracy, nor will it make you more or less likely to overfit.

gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • 5
    Totally agree with @gung. Please note, however, that this applies to OLS regression (the topic of this question, as I understand it) but may not apply to other algorithms for prediction. For example, if one were to use a lasso regression to predict weight from height and age, the scale of the variables will affect the model's calibration / performance. This effect is because there is a penalty for having coefficients with higher absolute values. As the scale of the variables affects the scale of the coefficients, the scale will similarly affect the lasso's calibration. – shf8888 May 31 '15 at 17:48
  • I am not quite sure if I understood this correctly: "penalty for having high coefficients with higher absolute values". If I look at this lasso example: http://pmtk3.googlecode.com/svn-history/r1340/trunk/docs/demoOutput/Variable_selection/lassoPathProstate_01.png Larger coefficients are penalized but the do not shrink to zero directly, how the smaller coefficients do. I mean, we do penalize but we do not penalize explicitly coefficients with higher absolute value, do we? – Anni Jun 02 '15 at 17:07
  • @Anni, shf8888 & Roger Fan are correct. I assumed you were referring to standard (OLS) regression (which still looks correct from context). In that case, the units make no difference. There are some algorithms (eg LASSO), that take the values into account. The LASSO penalizes (shrinks towards 0) the estimated betas, but the amount shrunk will differ based on their magnitude. Since the beta for a measurement in millimeters will be larger that a beta for meters, the relative shrinkage will differ depending on the units used. – gung - Reinstate Monica Jun 02 '15 at 17:15
  • As a result, it is commonly recommended to standardize your variables prior to using those kinds of algorithms. If you do standardize, it again won't matter what the original units were, but if you did not standardize, the units would matter. – gung - Reinstate Monica Jun 02 '15 at 17:16
  • To answer your explicit question, all betas are penalized in LASSO, but they are not penalized equally. The smallest are penalized the most. Thus, changing between mm & m will affect the amount that variable is de-emphasized by the LASSO. – gung - Reinstate Monica Jun 02 '15 at 17:18
  • @gung (quite late comment but still.. ) Why would a beta for a measurement in millimeters be larger than a beta for meters? Let say our output variable is in dollars and input variable is size in meters, beta for size is 300 (units of beta is: dollars / meters, formula: 300 (dollars / meters) * size (meters) ). By converting meters into centimeters: 300/100 = 3. So, 3 is my new beta with units dollars/cm, formula: 3 (dollars/cm) * size (centimeters). In fact, beta for centimeters turned to be smaller than a beta for meters. – Anni Apr 04 '16 at 14:11
  • @Anni, I'm not sure if I follow your example, but let's say that X is dollars & Y is meters, & that \$1 will buy you 1m of lumber ($\beta_1 = 1$), then \$1 will buy you 100mm of lumber ($\beta_1 = 100$). Alternatively, let's say that Y is dollars & X is meters, & that you can sell 1m of lumber you have chopped for \$1 ($\beta_1 = 1$), then you can sell 1mm of lumber for only 1 cent ($\beta_1 = 0.01$). In short, *nature is indifferent to units of measurement*; the betas will be scaled appropriately, but they will be scaled up or down depending, in part, on whether they are X or Y. – gung - Reinstate Monica Apr 05 '16 at 00:34
4

As @shf8888's comment notes, @gung's answer only applies to (simple) linear regression. In principle the units should not matter, but in other regression models this may not be automatic or easy to accomplish.

For instance, consider a simple 2D kernel regression, where

$$ \hat{E}(Y | X = x) = \frac{\sum_{i=1}^n K_h(x - x_i)y_i}{\sum_{i=1}^n K_h(x - x_i)}$$ and $$K_h(x - x_i) = K\left( \frac{x - x_i}{h} \right)$$

Here, if you change the units of $x$, the regression output will only stay the same if you also change the bandwidth parameter $h$ accordingly.

In this example it is fairly easy to keep the results the same under unit changes, but if you imagine the higher-dimensional version of this model then it's easy to see how it could become difficult to adjust.

In more complex models, it may be so difficult to maintain this equivalence that the choice of units actually becomes an important part of model tuning and analysis.

Community
  • 1
Roger Fan
  • 223
  • 2
  • 6
2

Shifting the regressor will not change the slope but the intercept.

$Y_{i} = \beta_{0} + \beta_{1}X + \epsilon_{i} = \beta_{0} + a\beta_{1} + \beta_{1}(X-a) + \epsilon_{i} = (\beta_{0} + a\beta_{1}) + \beta_{1}(X-a) + \epsilon_{i} = \beta^{*}_{0} + \beta_{1}(X-a) + \epsilon_{1}$

ie; shifting the reggression (for example height) will change the intercept only

Scaling the regressor will change the slope but not the intercept.

$Y_{i} = \beta_{0} + \beta_{1}X + \epsilon_{i} = \beta_{0} + (\beta_{1}/a) (X*a) + \epsilon_{i}$

luchonacho
  • 2,568
  • 3
  • 21
  • 38
B K
  • 141
  • 1
  • 6