3

I know it is common to include first order ($X$) and second order ($X^2$) terms in the same regression, but can I also include other terms in the same regression? For example, my variable selection techniques suggest I should regress the model:

$\hat{Y}_i=\hat{\beta_0}+\hat{\beta_1}\frac{1}{X_i}+\hat{\beta_2}log(X_i)$

Is this a reasonable thing to do? It looks like interpretation will be harder.

Raymond Hancock
  • 73
  • 4
  • If a close approximation to the actual (but unknown) function that generated the data was in fact "y = B0 + B1/x + B2 * log(x)", then fitting that equation and data would give good fitting results - in addition to a smooth looking plot through the center of the data and a normal-looking distribution to a histogram of the regression errors. – James Phillips Dec 21 '19 at 00:42
  • 1
    I don't see any reason why you couldn't try to fit such a regression. You may run into an issue if the transformed variables are highly correlated. It is really no different from doing polynomial regression. But you should want to have some reason for picking the model. – Michael R. Chernick Dec 21 '19 at 00:44

2 Answers2

2

Having several predictors that are each functions of a single variable do occur in more situations than just fitting polynomials; indeed numerous examples can be found in various questions on site; I'll mention a few.

  • See the discussion here under "other functions of $x$", and the fairly generic discussion here

  • See the model with multiple sin and cos components here

  • see any of a number of examples on site of regression splines. For example they are discussed here, though that's not a great introduction; perhaps see some of the introduction of ideas here and then the mention of the cubic spline basis functions here; that might help.

  • Getting a bit closer to your specific example, there's an example of fitting a particular case of a Hoerl curve (which crops up in physics a bit) -- $E(Y|x) = ax^b\exp{(-cx)}$ here. This is essentially a scaled gamma density (in the example, it's a special case with $b$ set to $1$). If you take logs of both sides (with some handwaving about the error term), you get $x$ and $\log x$ as predictors.

  • Note that your example predictors would correspond to the log of an inverse gamma instead; this doesn't seem much of a stretch. As long as the model had some justification, the only thing I'd worry much about there would be that for some ranges of $x$ the predictors might be pretty highly correlated which could lead to multicollinearity issues.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
0

I do not see any reason why you should not do this. Maybe from pure statistics point of view this might be quite rare, but for data science the technique above is actually used and very useful to improve machine learning model.

I do not know if you have ever heard the term before but what you are doing above is called feature engineering. Not only that you can also do interaction between term (which is quite similar to polynomial regression) but functions could be more complicated. However, usually you might want to remove some feature otherwise you either get dimensionality problem or overfitting.

For interpretability of this model this might not necessarily be a problem. In real life relationship between dependent and independent variable is not necessarily clear and linear hence this is actually very helpful and on top of that suppose one of the feature is actually a good predictor, it actually adds another fun question "why does this happen?".

Yohanes Alfredo
  • 290
  • 1
  • 6