0

So I am trying to use RF recursive feature elimination to extract the most predictive features from my data-set. I've gotten the code to run fine and it gives me a nice table of rankings. However, when I try re-running it it gives me a whole other set of most predictive features than it did the first time. I'm at a loss here guys. Why could this be?

I've done this both with K-fold cross validation and without and get the same problem.

My variables are ordinal so I scale them with MinMaxScaler, which should be correct?

I've tried both using a train test split with the same random seed and without still I keep getting different features each time I run it.

So essentially its all random and meaningless. Am I missing something here? Or am I doing something wrong?

Sincerely

P.S I don't know if I should post my code or not. If so let me know and I'll edit it in. But the code seems fine everything runs I just feel Im missing some step or something that I am doing wrong.

T.E.G.
  • 1,676
  • 6
  • 16
  • 29
Thomas Quick
  • 1
  • 1
  • It would appear your variables are mostly junk, hence the random selections. Although if you use the same seed you should always get the same result. – user2974951 Sep 17 '18 at 11:49
  • I hear you, but I'm using a small subset of my data where my inputs are question on the Beck anxiety inventory and it should be highly comorbid with depression which is my outcome according to the literature. Could it be that my scaling isn't the same for each time i run it? Is there a way to "set a seed" for the MinMaxScalar? – Thomas Quick Sep 17 '18 at 12:11
  • Oh and I am doing MICE imputations for the missing values. So could this also be a cause? – Thomas Quick Sep 17 '18 at 12:13
  • Have you looked at "[Boruta](https://www.jstatsoft.org/article/view/v036i11/)"? They have had to ask and answer a number of these questions and implement the fixes in code. – EngrStudent Sep 21 '18 at 16:03
  • Yeah actually when I use Boruta, it works perfectly. However, my supervisor wanted me to use recursive feature elimination so I guess I would have to justify using Boruta instead. Does it work in a similar fashion? Maybe that's off-topic and I should just go read up on it! Anyway thanks for the suggestion! :) – Thomas Quick Sep 21 '18 at 16:44

1 Answers1

1

Welcome Thomas,

well first of all: if you use the gini impurity-based feature importance measures you should not need scaling at all. This is due to the definition which only considers how well classes can be seperated along a certain feature dimension. You can find it here.

Apart from that, I see potentially two reasons for your observations:

Firstly, how many features do you have? If you have a very huge set of sparse, independent variables, you could end up with different important features for each bootstrap sample.

The second reason could be that you did not specify a minimum number of instances per leaf. If this equals 1 you can easily overfit on all features which are very sparse in the tree. And since the bootstrap samples are random, you can end up with different "top-features". This is especially true if the number of trees is very small.

Maybe there are other reasons, but those come to my mind first.

How to solve those potential problems:

1) Increase the number of trees.

2) Increase minimum number of instances per leaf.

I hope that helps.

Elmar Macek
  • 54
  • 3
  • Thanks for this answer Elmar. I don't quite understand the Gini impurity-based feature criterion. I'll try to look into it. The default criterion for Random Forest is set to gini and not entropy so I guess that means I should skip feature scaling? Would this also be the case for continous and categorical (which I would one-hot encode) variables? I'm running GridSearchCV and hopefully that will solve my problem where I adress both of your recommendations! Thanks for taking the time to explain this! – Thomas Quick Sep 20 '18 at 15:58
  • 1
    You are welcome Thomas. :) To the best of my knowledge gini and entropy are nearly the same and will in most cases lead to the same trees, but the latter involves the computation of the logarithm. Scaling is actually fine, especially if you wanna swap in other models without changing the preprocessing. You will not hurt the forest by doing so but it is not neccessary. Please let me know, if this solves the randomness. If not, I would think again and deeper with a little more information from your side. Best! ( Btw, if you feel this helped you i would be happy if you voted the answer. ) – Elmar Macek Sep 20 '18 at 17:57
  • 1
    Thanks Elmar, I voted your answer but since my reputation is still so low it won't show publically but it is recorded. Im still waiting for the GridSearchCV to finish running and I will get back to you as soon as I get my results to see if the problem is fixed. All the best! – Thomas Quick Sep 21 '18 at 13:31