1

I want to build a statistical model with the aim of answering what is the effect of x1 on y. Even though my aim is statistical inference, I'll keep a separate test set.

Let's say I estimate the model and it 'looks good', e.g. high adjusted R^2, but it does a horrible job on the test set in terms of prediction performance.

How should I interpred this? Should I trust the coefficient estimate of a model that does a poor job in terms of prediction performance?

Viðar Ingason
  • 407
  • 2
  • 10
  • I am not aware of meaning of test set. Why do you want that the two should match ? –  May 06 '19 at 00:21
  • 1
    Prediction and explanation are different goals. – Isabella Ghement May 06 '19 at 02:05
  • There is a good CV Q&A on the topic of whether cross-validation and causal inference go together: https://stats.stackexchange.com/q/3893/241093. – AlexK May 06 '19 at 03:28
  • SubhashC.Davar, if I have 1.000 observations I might build the model using only 800 observations and keep 200 for testing the accuracy of the model, (I know this is not a standard procedure, so that's why I'm asking). IsabellaGhement, I am aware of that. But my question is, should I feel comfortable in stating that the effect of x1 on y is b1 when I know that the prediction performance of the model is really poor. – Viðar Ingason May 06 '19 at 12:55

1 Answers1

0

You should not base conclusions you make about the estimates of regression coefficients on the predictive performance of your model. You can have a model with a low $R^{2}$ but that still produces an unbiased estimate of the relationship you are studying.

Whether a coefficient estimate is unbiased and represents the "true" relationship between $X_{1}$ and $Y$ depends on whether your model controlled for all confounding factors (variables that are correlated with both $Y$ and $X_{1}$ - such as years of work experience, years of schooling, and seniority of job position being confounding variables in the relationship between age and income).

On other hand, if you wanted to build a regression that predicts $Y$ well, it would be important to include variables correlated with $Y$ even if they are not correlated with $X_{1}$. With the income example, if you predicted income as a function of only the four variables I mentioned above, you would likely obtain a low $R^{2}$ because those variables do not explain all of the variation in income. The country or state you live in, the industry you work in, and even race and gender are good predictors of income, but those variables are probably not correlated with age, so they are not needed in a model whose goal is causal inference of the relationship between age and income.

(Also, $R^{2}$ and predictive performance can be a function not just of what predictors you include but what functional form your outcome/target variable takes.)

AlexK
  • 1,007
  • 4
  • 11