6

i want to make classification with naive Bayes. I have got about 100 Features. Numerical ones as well as categorical ones. Since i want only the most relevant ones to be included for the classification task i want to find them with some kind of feature elimination. My question now is the following: what is the method to use that for (paper/reference?!) and is this method implemented in some sort of software package. Since i use R i would especially prefer some R package.

Thanks in advance for your help!

user3008056
  • 101
  • 1
  • 1
  • 5
  • Something in here might help you: http://stackoverflow.com/questions/20600388/how-to-improve-feature-selection-for-naive-bayes – shadowtalker Jun 25 '14 at 05:44
  • @ssdecontrol thank you! Indeed there was a hint that WEKA supports exhaustive search to select the subset of features that performs best. Does anybody know if R also supports such a exhaustive search? – user3008056 Jun 25 '14 at 05:54
  • There's an R package called `FSelector` that seems to have a method for it, but I've never used it. It also hasn't been updated since February 2013. I'm surprised that R doesn't have a better-developed base of ML tools. Truth is, R supports anything you can program into it, it's just a matter of doing it. The only other thing I can offer is that you need to figure out a way to compute gain ratios efficiently; ML isn't something I know a whole lot about. Maybe some info here: http://stackoverflow.com/questions/17844520/calculating-information-gain-ratio – shadowtalker Jun 25 '14 at 07:29
  • As of 2013 at least, it seems at least a few people are writing code from scratch in C++: www.iaeng.org/publication/WCE2013/WCE2013_pp1549-1554.pdf – shadowtalker Jun 25 '14 at 07:35
  • @ssdecontrol Thank you very much... i found `exhaustive.search()` in `FSelector`. This may do what i want to do. I will try to make some example with toy data and eventually post it as one kind of solution. – user3008056 Jun 25 '14 at 08:07

2 Answers2

5

There are two different routes you can take. The key word is 'relevance', and how you interpret it.

1) You can use a Chi-Squared test or Mutual information for feature relevance extraction as explained in detail on this link.

In a nutshell, Mutual information measures how much information the presence or absence of a particular term contributes to making the correct classification decision.

On the other hand, you can use the Chi Squared test to check whether the occurrence of a specific variable and the occurrence of a specific class are independent.

Implementating these in R should be straight-forward.

2) Alternatively, you can adopt a wrapper feature selection strategy, where the primary goal is constructing and selecting subsets of features that are useful to build an accurate classifier. This contrasts with 1, where the goal is finding or ranking all potentially relevant variables.

Note that selecting the most relevant variables is usually suboptimal for boosting the accuracy of your classifier, particularly if the variables are redundant. Conversely, a subset of useful variables may exclude many redundant, but relevant, variables.

Zhubarb
  • 7,753
  • 2
  • 28
  • 44
3

The R package caret (**C**lassification **A**nd **R**Egression **T**raining) has built-in feature selection tools and supports naive Bayes. I figured I'd post this as an answer instead of a comment because I'm more confident about this one, having used it myself in the past.

shadowtalker
  • 11,395
  • 3
  • 49
  • 109
  • 1
    yes you are right. I have already found the function `rfe()` in `caret`. But is this the best way to handle it? Are there other functions more recommended? Does anybody have further insights? – user3008056 Jun 25 '14 at 08:06