4

Please pardon me if you find this question very silly but this doubt has been troubling me for some time now whenever I want to run a regression.

I am working on SAS. I have a dataset which has 24,000 observations, and there are about 50 independent variables. There are no missing values and/or outliers. Dummy coding for categorical variables is also done. So, data preparation is complete. Now, when I run a regression model on this dataset, there are a few variables (8 variables) for which p-value is > 0.05 i.e. these variables are insignificant.

My question is what next? Do we remove these variables from the final regression equation? So, instead of having 50 independent variables, we'll have 42 independent variables (42 Beta coefficients + 1 for constant). Or do we need to remove one of these insignificant variables and re-run the regression model to see if there's any previously insignificant variable becomes significant now?

abhi1495
  • 43
  • 1
  • 1
  • 5
  • 6
    What is the problem? What do you want to do? Will the model be used for prediction in the future, and avoid measuring those 8 variables will save money? If not, I cannot see any problem with just leaving those 8 variables in the equation. – kjetil b halvorsen Sep 01 '15 at 20:47
  • 2
    See [Should covariates that are not statistically significant be 'kept in' when creating a model?](http://stats.stackexchange.com/q/66448/17230). – Scortchi - Reinstate Monica Sep 02 '15 at 09:36

1 Answers1

0

There are several ways of tackling this problem. Most people would use a variable selection technique for choosing which variables to keep and which to discard. There are a few things you have to consider before doing this:

1) Do you have a limit on the number of covariates you want to explain the change in the response? (i.e. you only want 10 variables at most)

2) Many of the variables can be strongly related, so that they don't really provide linearly independent information.

Thus, the goal of any variable selection is to removed unnecessary covariates with negligible contribution and to removed correlated covariates so that the remaining covariates provide as much independent information about the response as possible.

There a ton of details that go into variable selection, and more than can be explained in one answer. But, there are three popular types of variable selection: Forward Selection, Backward Removal, and Stepwise Regression (combines both forward and backward). Stepwise Variable Selection is generally thought of being the best (from my knowledge).

So how does one do this? It depends on your software and I do not know to do it within SAS. I can write you the basic steps for completing it, but you'll have to figure out some of it yourself in SAS.

Another thing you have to consider: Just because a variable's p-value is > 0.05, is it realistic to remove it? In some cases, we can remove variables because they are insignificant in explaining the response. But in some cases, even insignificant variables must be kept.

Probably the easiest way, but not necessarily the best, would to remove the most insignificant variable one at a time until all remaining variables are significant. Hope this helps!

Jake
  • 340
  • 2
  • 11
  • 5
    Forward, backward & stepwise variable selection are invalid. They should not be used or recommended. For a conceptual overview, it may help to see my answer here: [Algorithms for automatic model selection](http://stats.stackexchange.com/a/20856/7290). – gung - Reinstate Monica Sep 01 '15 at 21:20
  • @gung While I agree with you that maybe they aren't the best methods or recommended in all cases from some of my own research, they do have some use. That or I just wasted a semester of grad school learning about them haha. – Jake Sep 01 '15 at 21:28
  • 3
    To be honest, I think your having spent a semester learning about them was largely a waste. It is possible to use them validly by incorporating them into a cross-validation scheme, shrinking the coefficients & using special adjustments so that the p-values will be valid. But few people know about those methods, & they are hard to do. There are other methods that are more straight-forward to select variables without overfitting or invalidating your hypothesis tests. – gung - Reinstate Monica Sep 01 '15 at 21:32
  • Thanks Jake and gung for your valuable inputs. Since, I am not an expert on regression and I don't want R^2 of perfectly 1, Stepwise regression should also work for me. All the help is appreciated :) – abhi1495 Sep 03 '15 at 06:09
  • 1
    Just to clarify, make sure you aren't using R^2 as a model selection criterion. Because of the nature of R^2, it will also go **up** if you add more covariates, even if they don't add any new significant information. R^2 adjusted, or even AIC would be a lot better of a selection criterion. And @gung, we didn't spend the whole semester looking at stepwise regression, but it was the only method introduced as a model selection tool – Jake Sep 04 '15 at 18:50
  • You still have the same problem with adjusted R^2 & AIC. I discuss that in the linked answer. – gung - Reinstate Monica Sep 04 '15 at 19:09