0

I am looking for the best $\alpha$ (=ratio between L1 and L2 penalty) and $\lambda$ (=penalty strength) for my elastic net regression model, using the R package glmnet. I have found my $\alpha$ by cross-validation using cv.glmnet & fixed foldids, following the Glmnet vignette and Stackexchange question (see code below).

Short version: I don't know if I have to tune for $\lambda$ seperately using the tuned $\alpha$ OR if I should directly extract the lambda.min from the same cv.glmnet object?

Longer version: After I tune for my $\alpha$ following this code:

#library("glmnet")

#Tune for alpha (L1/L2):
foldid=sample(1:10,size=length(y),replace=T)
alphalist <- seq(0,1,by=0.1)
cv_alpha <- lapply(alphalist, function(a){
  cv.glmnet(x, y, foldid=foldid, alpha=a, family="gaussian")
})
for (i in 1:11) {print(min(cv_alpha[[i]]$cvm))}

#cv_alpha[[9]] results in the minimum cvm and is chosen for further analyses

...should I directly extract the lambda.min from the same cv.glmnet object I used for tuning for $\alpha$ by calling it with cv_alpha[[9]][["lambda.min"]] when extracting the coefficients, as in the code below:

##first run the glmnet
set.seed(1010)
elast = glmnet(x, y, alpha=0.8, family="gaussian")
## extract coefficients at specific lambda as determined in cv_alpha
beta_elast <- coef(elast, s = elasticnet[[9]][["lambda.min"]])

OR is it more advisable to cross-validate for the $\lambda$ seperately with a new cv.glmnet at the determined $\alpha$?

lambdas <- 10^seq(2, -3, by = -.1)
cv_lambda <- cv.glmnet(x, y, alpha = 0.8, lambda = lambdas)
beta_elast <- coef(elast, s = cv_lambda$lambda.min)

Edit: Added a short version of the question further up and edited what I mean with $\alpha$ and $\lambda$

midas
  • 1
  • 1
  • Both hyperparameters should be optimized simultaneously. So, you'd consider pairs of values $\{(\alpha_1, \lambda_1), \dots, (\alpha_n, \lambda_n)\}$. For each pair, use cross validation to estimate the generalization performance. Then, keep the pair with best performance. – user20160 Feb 02 '21 at 18:24
  • @user20160 Thank you, that is very helpful! So to consider pairs of values would it be enough if I simply adapt my code from above to`elasticnet2 – midas Feb 03 '21 at 15:26
  • Can't comment on the implementation, as I'm not an R user – user20160 Feb 03 '21 at 21:22
  • Alright, but thank you for your reply! – midas Feb 04 '21 at 11:46

0 Answers0