1

Assume we have 500 predictors and one response. Can we perform a univariate regression on each pair Y-X and then select the predictors that have the highest R-squared and p<0.05? After that, we can use the selected predictors in multivariate combinations or step regression etc.

I have not found an answer why this is not advisable and I cannot mathematically prove why intuitively this sounds wrong.

I know that selecting the top-20, for example, predictors based on univariate R-squared does not guarantee that a multivariate model created by any combination of them will be better than a multivariate of the next 20 or the bottom 20.

Jespar
  • 564
  • 4
  • 12
  • Stepwise approaches have a lot of problems, which are explained well here: https://stats.stackexchange.com/a/20856/121522 – mkt Jun 05 '18 at 08:56

1 Answers1

0

The approach of fitting a separate simple univariate linear regression model for each predictor is not entirely satisfactory as each of the 500 regression equations (one for each predictor) will ignore the other predictors in forming estimates for the respective regression coefficients.

If the predictors are correlated with each other this can lead to further problems.

Also , by fitting a separate univariate linear regression model, we may get a small p-value(i.e. the predictor is significant) for some of the predictor's coefficients. But , when a multiple linear regression model is fitted which includes many of the predictors , some of the predictor coefficients may have a large p-value(i.e. the predictor is insignificant).

Now , it can be the case that a predictor's coefficient is having a large p-value when present in a multivariate regression model ,while, in the case of a univariate linear regression model , was having a smaller p-value. Hence , this can also cause problems in interpreting the important predictors.

Techniques such as mixed selection , can help overcome the problem of changing p-values for predictor coefficients.

Faraz Gerrard Jamal
  • 111
  • 5