2

I am working on a variable selection problem where there are a dependent Y and around 10 predictors X_1 to X_10. I would like to pick the three best predictors. The predictor list is already materially censored to match the domain I am working in and I have already reduced dimensionality with LASSO. Data is definitely not normal but the exact distributions are not static. The predictors are not all on the same scale, some are monetary values, others are %. The dependent Y is always monetary.

I would like to show a relative rank ordering of the shortlisted variables. I am not fitting a multiple linear regression but even if I did, the relativity of the coefficients would be misleading if I do not transform.

I am running a AIC-minimizing variable selection from regsubsets (package leaps) in R. AIC is different in each run - but this is expected.

My problem is that:

If I don't standardize, predictors X_1, X_3 and X_5 are selected.

If I standardise, predictors X_1, X_7, X_10 are selected.

If I normalise, predictors X_3, X_5, X_10 are selected.

What is the best approach?

J. Doe.
  • 143
  • 6
  • I just saw this post, http://stats.stackexchange.com/questions/32649/some-of-my-predictors-are-on-very-different-scales-do-i-need-to-transform-them which alerted me to the fact that I have set intercept=FALSE. If I re-instate it, the variables shortlisted are the same. – J. Doe. Nov 22 '16 at 11:05
  • 1
    What you mean by "normalise"? if that means transform variables with a view to getting them closer to normality, it's not a surprise that some predictors may appear distinctly better afterwards. But it's not the normality of distributions that matters, so much as the relationship between the response and each predictor given the others. I'm surprised that standardisation makes a difference, assuming that means (value $-$ mean) / SD. (Conversely, if you doubt that transformation may affect the chosen model, why do it?) – Nick Cox Nov 22 '16 at 11:59
  • 2
    "the exact distributions are not static": please explain what that means. – Nick Cox Nov 22 '16 at 12:00
  • normalise == rebase to mean=0 and var=1; standardize==rebase to mean=0 only.. This variable selection is to impose sparsity among the 10-variable subset across multiple data subsets (90 to be exact). I want to avoid panel regressions. This is what I mean when I say the distributions are not static; different Y~X variables are fed into the AIC minimizing LEAPS library. – J. Doe. Nov 22 '16 at 12:11
  • Importantly, I believe I have a viable solution now; but I would appreciate if someone criticized it. I normalize variables by data subset (i.e. 90 normalizations) and I run this. leaps_out – J. Doe. Nov 22 '16 at 12:13
  • 2
    I'd advise that those definitions are not universal across statistical science. I would even assert that your "normalise" is usually called "standardise" and your "standardise" would be called "centring on the mean" or some such wording. If you can find authoritative texts supporting either definition, I would be interested. Otherwise, on what interests you, I can't comment further, as I don't use the selection procedures concerned. I am puzzled that linear transformations make any difference at all. – Nick Cox Nov 22 '16 at 12:16
  • The first comment clarifies why I got wrong results with variable selection - I was not including the intercept. On that basis the question is already answered. Thank you for your help/ – J. Doe. Nov 22 '16 at 12:20

0 Answers0