1

I am performing an elastic net regression on my data n = 34, p = 46

I first built the model using the "caret" package with the cross validation method to set the optimal alpha and lambda parameters

data.scale <- as.data.frame(scale(data)) 
set.seed(123)
model <- train(
  value ~., data = data.scale, method = "glmnet",
  trControl = trainControl("cv", number = 10),
  tuneLength = 50)

Then I extracted the beta coefficients using the best lambda parameter

model$bestTune
coef(model$finalModel, model$bestTune$lambda)

and then, I tested the model performance on the entire dataset and calculated the RMSE and R2

x.test <- model.matrix(value ~., data.scale)[,-1]
predictions <- model %>% predict(x.test)
data.frame(
  RMSE = RMSE(predictions, data.scale$value),
  Rsquare = R2(predictions, data.scale$value)
)

Now I am trying to calculate the pvalues by bootstraping the model with the function boot.glmnet from the package "hdrm" with the following procedure

# divide explanatory variables and response variable in two separate dataframes
data.x <- data.scale %>%
  dplyr::select(-value)
data.y <- data.scale %>%
  dplyr::select(value)

# calculate confidence intervals
CI <- boot.glmnet(data.x, data.y$value, 
            alpha = 0.05,
            lambda = model$bestTune$lambda,
            B = 1000,
            bar = T)

in CI results I am getting zeros for all confidence intervals lower and upper. I am now wondering where I am wrong. Is it a conceptual or scripting mistake? or maybe both? I would be very happy if anyone could clarify me this. I can provide the data if needed Thanks

Alberto Pascale
  • 11
  • 4
  • 1
    Why do you want p-values? – Dave Jul 20 '20 at 23:41
  • I guess because I want to test the reliability of the coefficients and then of the ranking of my variables – Alberto Pascale Jul 20 '20 at 23:47
  • 1
    I'm a bit surprised boot.glmnet exists, since https://amstat.tandfonline.com/doi/abs/10.1198/jasa.2011.tm10159#.XxYx15MzZoM argues that the bootstrap doesn't work for lasso estimators and https://cran.r-project.org/web/packages/bcaboot/vignettes/bcaboot.html describes a bias correction that boot.glmnet doesn't seem to use – Thomas Lumley Jul 21 '20 at 00:10
  • Thank you for your answer @ThomasLumley, I was not aware of this bias and I will give a try to this package – Alberto Pascale Jul 21 '20 at 21:53

1 Answers1

4

Coding issues are off-topic on this site, but the statistical issues about determining confidence intervals in the LASSO part of elastic net deserve some comment. In brief, this is not an easy problem. The hdrm package does not appear to be on CRAN or externally vetted; its boot.glmnet() function is an overly simple (mis?)application of the bootstrap as the comment from Thomas Lumley notes.

This page and its links provides an introduction to the difficulties with estimating CI and p-values with LASSO. For example, LASSO's choice among a set of correlated predictors might differ from one bootstrapped sample to another. How do you want to think about CI for a predictor that is sometimes present and sometimes omitted from a model? For some types of models it is possible to estimate CI for predictors, but that requires substantial care and thought.

With LASSO and elastic net the primary consideration is usually predictive performance. You might consider developing models on multiple bootstrapped samples of your data and evaluating predictive performance against the full original data set as a way to estimate the reliability of your modeling process.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • +1. At https://stats.stackexchange.com/a/446205/919 I illustrate some techniques for tracking and graphing a situation where the number of estimated parameters can vary from one iteration to the next. – whuber Jul 21 '20 at 12:19
  • Thank you EdM. I will reconsider about bootstraping for CI. I was using elastic net regression because it seemed the easiest method to get answers from my data. Considering your thoughts and reading @Tom Wenseleers answer on (https://stats.stackexchange.com/users/27340/tom-wenseleers) L0 regression works better than elastic net for features selection. I also tried to perform this analysis but I still have some doubts on how it works, so I switched back to elastic net. Going back to the last part of your answer, I am not sure how different is it from the LOOCV to evaluate model performance – Alberto Pascale Jul 21 '20 at 22:22
  • @AlbertoPascale LOOCV is a very deterministic approach that often shows high variance, as the models evaluated aren't really that different and you only evaluate each fold on a single case. Developing models on bootstrapped samples of the same size as the original data and testing them on the entire data set can be a more reasonable approximation to the process of sampling from a population, developing a model on the sample, then testing it on the population. The bootstrap principle is that the bootstrapped samples are to the original sample as the original sample is to the population. – EdM Jul 21 '20 at 23:03
  • @AlbertoPascale the L0 regularization discussed [here](https://stats.stackexchange.com/a/423174/28500) isn't that hard to understand. LASSO (and elastic net in its predictor-selection part) uses an L1 penalty, on the sum of the _magnitudes_ of the coefficients of the included predictors. The L0 penalty is on the actual _number_ of included predictors. That said, the L0 advantages may mostly be in very large data sets. With 34 cases you will only be selecting a handful of predictors in any event, maybe 2 or 3 with LASSO or a few more with elastic net. – EdM Jul 22 '20 at 14:55
  • @EdM: (obvious +1) I was reading your comment but then I came across [Minnier et al. (2011)](https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm10382) so I actually think if the whole LASSO procedure (i.e. with the dynamic estimation of $\lambda$) is bootstrapped I think it can work. I have seen some further critique on it regarding its second order correctness by [Das et al. (2019)](https://projecteuclid.org/journals/annals-of-statistics/volume-47/issue-4/Perturbation-bootstrap-in-adaptive-Lasso/10.1214/18-AOS1741.short) but I think in general the approach... holds! – usεr11852 Feb 23 '21 at 02:41