How to gain knowledge from dataset using regressions in R

Question

I have some data: outcome is satisfaction and there are four predictors, three continuous (age, weight, height) and one factor, graduated high school or not.

In R, I have uploaded the data set, and set $X1$ for age, $X2$ for weight , $X3$ for the factor and $X4$ for height.

I want to know if there is evidence that graduating high school has an effect on satisfaction.

I know that I can not simply look at lm(y~x3), because I need to consider all the other possibilities. So how do I take all of these into account? How many models must I check? What is the general approach to this?

Also, would I need to consider any and all possible interactions?

Call:
lm(formula = y ~ x1 + x2 + x3 + x4)

Residuals:
    Min      1Q  Median      3Q     Max 
-18.506  -5.096   1.306   4.738  28.722 

Coefficients:
            Estimate     Std. Error      t value       Pr(>|t|)    
(Intercept) 140.1689      8.3191          16.849      2.77e-13 

x1           -1.1428     0.1904          -6.002       7.22e-06 

x2           -0.4699     0.1866           -2.518     0.0204 

x3yes         2.2259     4.1402            0.538     0.5968    

x4            1.2673     1.4922     0.849      0.4058    


Residual standard error: 9.921 on 20 degrees of freedom
Multiple R-squared:  0.8183,    Adjusted R-squared:  0.7819 
F-statistic: 22.51 on 4 and 20 DF,  p-value: 3.611e-07

Answer 1

The most important thing to do is for you to check if the model makes sense. You have fit a linear model to three continuous predictors, you need to make sure that it makes sense to do so You should look at scatterplots of age, height, and weight against y, and adjust the fits of these predictors if needed.

Assuming fitting these predictors linearly is reasonable, fitting the full model with all four predictors is a sensible thing to do.

You have only 25 data points. If you go on a long search through the space of all models (adding and removing variables) you have an extremely high risk of false positives. So, I don't think there is much need to backwards select out variables; if you wish to do so, make sure you use cross validation to make sure doing so improves the fit of the model to unseen data.

The same thing applies to a search for interactions, you have little data, and you are running a large risk of false positives.

If you wish to make inferences using the estimated confidence intervals, you should additionally check a plot of the residuals vs. the fitted values of the model and make sure you do not see any patterns. You're looking to see if they look like they could have been drawn from a normal distribution with constant variance. If this looks reasonably consistent with yor data, then you can make inferences about the graduation parameter using the linear model

Coefficients:
            Estimate     Std. Error      t value       Pr(>|t|)    
(Intercept) 140.1689     8.3191          16.849        2.77e-13     
x1           -1.1428     0.1904          -6.002        7.22e-06     
x2           -0.4699     0.1866          -2.518        0.0204     
x3yes         2.2259     4.1402           0.538        0.5968        
x4            1.2673     1.4922           0.849        0.4058    

The x3 variable measures graduation, and its parameter lies well within the error of its estimation. So, given that everything above checks out, the data you used to train the model is not inconsistent with the effect of graduation being indistinguishable from noise.

Thanks so are we really able to judge this just from fitting the full model?

As long as all the caveats are met, I do think the best way to go about this is to fit the full model, and make your inference from that. Like I said, any inference you draw from a model that does variable selection is likely to occurr by chance.

Another way to think about this is, if you go through a variable selection algorithm, the standard errors reported in the model are no longer correct, they are actually much larger than what is reported. To estimate the true standard errors of the parameter estimates under a selection / fitting procedure, you would need to use either nested cross validation or a bootstrap + cross validation. This would drive your data very, very thin, and incur a lot of variance (you are making lots of decisions, each has a chance to be wrong). Your standard errors would be enormous.

Answer 2

Because there can be dependencies between the predictor variables it is possible that say X1 looks significant when X2 is left out but because X1 and X2 are highly depend X1 may appear non-significant when X2 is included in the model. With four predictor variablesthere are 2^4 -1 possible non-empty models. As this is only 15, it is not difficult to look at all subsets. If the number of variables were much larger a step-wise approach should be adequate. If possible pick a model where all the coefficients are significant and if you have 2 highly correlated variables make sure that one is excluded.