0

Why is feature selection such a common topic, when one could simply use SelectKBest to find the optimal set of features? The only trouble would be to find the best K and choosing the best of the three available scoring functions.

Edit:

There seems to be consensus that automatic feature selection is to be avoided for a lot of reasons. As such, my question is: do selection functions like SelectKBest have any value? If not, should they be removed from scikit-learn?

Mixel
  • 133
  • 4
  • Please look at the discussion on [this page](https://stats.stackexchange.com/q/350587/28500) about best-subset selection versus LASSO, and on [this page](https://stats.stackexchange.com/q/20836/28500) about the difficulties posed by automated model selection in general. Also, consider how the number of possible groups of K features over which to search increases as the total number of features increases. Then please edit your question to focus on more specific questions you have that are still unanswered. – EdM Nov 18 '19 at 20:12

2 Answers2

2

The problem is more general than this.

Software packages (and not just SciKit - pretty much any package) do not provide tools just because they are useful and do not (in my experience) remove them because they are not useful. Tools get put into packages (and not removed) because:

  1. People want them. People ask for them. For commercial packages, meeting customer demand is part of making money. For free packages, it's still part of "making customers happy". People really want automatic methods for variable selection and many of them don't want to listen to the reasons this is bad.

  2. People want to write the code. The people who do the coding may not know the extent of problems with the method. Or the code may have been written before the problems were discovered.

  3. Removing code is problematic. If a person updates software and it then makes previous programs return errors, that person is likely to be very unhappy, regardless of whether the error is, really, a good thing.

Now, as to your precise question -- do these selection methods have any value -- well, maybe. For one thing, they can let you replicate earlier research. For another, well, in a similar context I read someone's reply (I forget who -- it might have been Ripley, but I'm not sure)

The only reason to do this is to demonstrate how wrong it is.

So, you could use a "select best" function to show that select best functions don't work very well.

Peter Flom
  • 94,055
  • 35
  • 143
  • 276
0

SelectKBest in just a wrapper, you may want to refer to this question: https://datascience.stackexchange.com/q/10773/85516

Personally when I am talking of feature selection, I am generally also implying feature engineering under the hood as I see it as the selection of the features that will be used to train the model (but maybe I'm wrong to do so). In this sense, SelectKBest does not provide ways to implement feature engineering.

Jeanba
  • 141
  • 3