16

So I've been playing around with SVMs and I wonder if this is a good thing to do:

I have a set of continuous features (0 to 1) and a set of categorical features that I converted to dummy variables. In this particular case, I encode the date of the measurement in a dummy variable:

There are 3 periods that I have data from and I reserved 3 feature numbers for them:

20: 21: 22:

So depending on which period the data comes from, different features will get 1 assigned; the others will get 0.

Will the SVM work properly with this or this is a bad thing to do?

I use SVMLight and a linear kernel.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
user3010273
  • 377
  • 1
  • 3
  • 9

2 Answers2

23

Replicating my answer from http://www.quora.com/Machine-Learning/What-are-good-ways-to-handle-discrete-and-continuous-inputs-together/answer/Arun-Iyer-1

  • Rescale bounded continuous features: All continuous input that are bounded, rescale them to $[-1, 1]$ through $x = \frac{2x - \max - \min}{\max - \min}$.
  • Standardize all continuous features: All continuous input should be standardized and by this I mean, for every continuous feature, compute its mean ($\mu$) and standard deviation ($\sigma$) and do $x = \frac{x - \mu}{\sigma}$.
  • Binarize categorical/discrete features: For all categorical features, represent them as multiple boolean features. For example, instead of having one feature called marriage_status, have 3 boolean features - married_status_single, married_status_married, married_status_divorced and appropriately set these features to 1 or -1. As you can see, for every categorical feature, you are adding k binary feature where k is the number of values that the categorical feature takes.

Now, you can represent all the features in a single vector which we can assume to be embedded in $\mathbb{R}^n$ and start using off-the-shelf packages for classification/regression etc.

Addendum: If you use Kernel Based Methods, you can avoid this explicit embedding to $\mathbb{R}^n$ and focus on designing custom kernels for your feature vectors. You can even split your kernel into multiple kernels and use MKL models to learn weights over them. However, you may want to ensure positive semi-definiteness of your kernel so that the solver doesn't have any problems. However, if you are unsure of whether you can design custom kernels, you can just follow the earlier embedding approach.

TenaliRaman
  • 3,542
  • 25
  • 26
  • Well, [this article](http://www.willmcginnis.com/2015/11/29/beyond-one-hot-an-exploration-of-categorical-variables/) is quite interesting regarding categorical features. It's not said that one-hot encoding is the best choice for categorical features is what I'm getting out of it. – Stefan Falk Sep 18 '17 at 16:55
  • 1
    this is an excellent answer, I read the link in @displayname comment and it is a useful comparison. From the article, it appears that binary encoding is the best, (not the one hot described in this answer) and quite simple as well) From the link " Binary: first the categories are encoded as ordinal, then those integers are converted into binary code, then the digits from that binary string are split into separate columns. This encodes the data in fewer dimensions that one-hot, but with some distortion of the distances." – shelbypereira Dec 11 '17 at 14:33
  • The article given by @displayname is a good article, but should not be taken at face value. The first thing to remember is that almost all ML methods work with either similarity or distance measure. The choice of encoding method directly influences how the distances or similarities are measured between two points. A 1 hot encoding says that an object of one category is similar to only itself or equivalently, it puts all categories in equal distance to each other. However, there are cases where certain categories are closer than others. In which case, a different encoding can help. – TenaliRaman Dec 12 '17 at 13:43
8

SVMs will handle both binary and continuous variables as long as you make some preprocessing: all features should be scaled or normalised. After that step, from the algorithms' perspective it doesn't matter if features are continuous or binary: for binaries, it sees samples that are either "far" away, or very similar; for continuous there are also the in between values. Kernel doesn't matter in respect to the type of variables.

iliasfl
  • 2,514
  • 17
  • 30