3

I have a data set with n=199 and p=130. I need to reduce the number of predictors for a regression. I did a Lasso regression but the numer of optimal variables changes dramatically (range from 6 to 41 predictors recommended by Lasso) when I change the seed number input. How could I determine the optimal set of variables to include in the model?

These are my different number of variables that i got.

variables
  [1] 24 21 29 24 21 21 24 33 24 33 24 39 21 21 24 21 29 21 29 29 24 29 29 29 24 18 29 24 21 39 39 21 21 29 33 39 24 41
 [39] 24 13 29 13 29 29 21 24 24 39 24 33 24 29 33 45 21 21 21 24 29 33 29 29 18 21 21 24 21 21 33 24 33 21 21 21 21 29
 [77] 41 24 39 21 21 24 21 39 29 21 39 21 41 21 33 21 33 33 21 33 21 24 21 29 21 21 21 29 24 21 21 21 29 29 24 21 24 21
[115] 21 21 21 33 18 21 39 29 29 21 42 21 13 24 33 21 39 24 29 33 21 41 29 24 42 33 41 21 21 21 13 24 21 24 24 39 24 41
[153] 50 33 21 24 24 18 29 24 39 21 21 24 33 42 21 21 24 29 24 21 24 24 24  6

My target is a dummy variable and my code is this

variables=rep(0) # vector with the numbers of variables with differente seed number
num.var<-0 #number of variable for any best lambda
iter<-0  #number of iteration
seed_input=500 # first seed number

while(iter <177){
seed_input=seed_input+floor(rnorm(1,10,2)) # change the seed number in diferent iteration
set.seed(seed_input) #use this seed number 
modelo<-glmnet(independent,dependent, family = "binomial", alpha=1) #fitting lasso regression
cv.modelo <- cv.glmnet(independent, dependent, alpha=1) 
best.lambda <- cv.modelo$lambda.min #saving best lambda #saving best lambda

    # Creating a vector with number of variables
n=dim(a)[1] 
    vars=rep(0)
    j=1
    for(i in 2:n){ 
      if(a[i,1]!=0){vars[j]=i
      j=j+1}
    }
    num.var[iter]<-length(vars) #save the numbers of variables in a vector
    iter=iter+1
    }

I hope that somebody can help me. Thanks.

Frank Josué Castillo Isisola
  • 65
  • 5
  • Please type your question as text, do not just post a photograph (see [here](http://meta.stats.stackexchange.com/a/3176/)). – gung - Reinstate Monica Feb 01 '17 at 01:05
  • I suggest putting the whole executable example. How did you get those numbers of variables? – Pere Feb 01 '17 at 01:09
  • Lasso is typically used to reduce the feature space of the model (ie number of attributes, number of data columns), not the number of samples. From the question, it is difficult to discern what is `n` (typically designates sample size), what is `p` and what you're trying to reduce. – Gene Burinsky Feb 01 '17 at 01:09
  • Cross-validation will tell you which set of variables yields the "best accuracy" or best out of sample prediction. – Gene Burinsky Feb 01 '17 at 01:10
  • Thanks to all for your awnser. @GeneBurinsky, n is the size of my sample and p is the number of variables. I need to reduce the number of variables but i don't understand why this number of variables changes with differents set seed. – Frank Josué Castillo Isisola Feb 01 '17 at 01:22
  • I cannot quite see where the seed is used. I don't think it is used for fitting the LASSO (even though you set the seed right before fitting), so you title does not make sense to me. Or perhaps the seed is used in cross validation? But then you could move the set.seed code line and edit your title accordingly. – Richard Hardy Feb 01 '17 at 06:29

1 Answers1

4

The reason your results change is because of the randomization used to cross-validate the estimates. In k-fold cross-validation, (ie what I think glmnet uses), each time cross-validation is run, a part of your data is randomly selected to be a training set and the remainder of the data is selected to be the test set. When you set different seeds, you are changing which parts of the data end up in the training and testing parts. As a result, at each different seed, glmnet fits the models to a different training set and cross-validation tests it on a different testing set. Consequently, the variables selected at each seed may vary.

Dirty ways around this: you could leave the seed alone or fit a large number of times on different seeds seeds and select the variable set that is chosen most often.

As mentioned by Gammer in the comments, leave-one-out cross-validation will not suffer the drawback from seed changes but may take a lot longer to run.

Check this cv question, the OP had a similar concern and the response may help answer your question of "how" do I select the best set.

Community
  • 1
Gene Burinsky
  • 338
  • 2
  • 8
  • Thanks a lot. Do you think that It happend because i have a small data set or it doesn't have influence? – Frank Josué Castillo Isisola Feb 01 '17 at 02:10
  • Your data are quite small and that may have something to do with it though distinct clustering within the data regardless of size could also influence this. I'm sure there are numerous other reasons that could cause this. It's difficult to tell as I'm not familiar with your data – Gene Burinsky Feb 01 '17 at 02:16
  • 1
    Yes, the random determination of folds is the reason for the inconsistent results, but note that this shouldn't affect leave-one-out cross validation, because the folds are deterministic in that case. – gammer Feb 01 '17 at 03:28
  • @FrankJosuéCastilloIsisola, Frank Harrell had some related answers here on Cross Validated (about what to do if LASSO is unstable), you may check them. – Richard Hardy Feb 01 '17 at 06:28
  • @RichardHardy, could you give me the link ? Thank you! – Frank Josué Castillo Isisola Feb 01 '17 at 14:50
  • [Here](http://stats.stackexchange.com/users/4253/frank-harrell) is his profile. But he has so many answers that perhaps it will be easier to simply search on Cross Validated using the relevant keywords, perhaps restricting the search within certain tags. – Richard Hardy Feb 01 '17 at 14:53