22

Recently I have learned that one of ways for finding better solutions for ML problems is by creation of features. One can do that by for example summing two features.

For example, we possess two features "attack" and "defense" of some kind of hero. We then create additional feature called "total" which is a sum of "attack" and "defense". Now what appears to me strange is that even tough "attack" and "defense" are almost perfectly correlated with "total" we still gain useful information.

What is the math behind that? Or is me reasoning wrong?

Additionally, is that not a problem, for classificators such as kNN, that "total" will be always bigger than "attack" or "defense"? Thus, even after standarization we will have features containing values from different ranges?

MrKadek750
  • 223
  • 2
  • 5
  • The practice of summing two features certainly doesn't represent "feature engineering" in general. – xji Jan 24 '18 at 22:40

5 Answers5

22

You question title and the content seems mismatched to me. If you are using linear model, add a total feature in addition to attack and defense will make things worse.

First I would answer why feature engineering work in general.

A picture is worth a thousand words. This figure may tell you some insights on feature engineering and why it works (picture source):

enter image description here

  • The data in Cartesian coordinates is more complicated, and it is relatively hard to write a rule / build a model to classify two types.

  • The data in Polar coordinates is much easy:, we can write a simple rule on $r$ to classify two types.

This tell us that the representation of the data matters a lot. In certain space, it is much easier to do certain tasks than other spaces.

Here I answer the question mentioned in your example (total on attack and defense)

In fact, the feature engineering mentioned in this sum of attack and defense example, will not work well for many models such as linear model and it will cause some problems. See Multicollinearity. On the other hand, such feature engineering may work on other models, such as decision tree / random forest. See @Imran's answer for details.

So, the answer is that depending on the model you use, some feature engineering will help on some models, but not for other models.

Ibraheem Moosa
  • 147
  • 7
Haitao Du
  • 32,885
  • 17
  • 118
  • 213
18

The type of model we are using might not be very efficient at learning certain combinations of existing features.

For example, consider your example where features are a and d, and we are using a decision tree to predict a binary outcome that happens to be $0$ if $a+d<0$ and $1$ if $a+d\geq0$.

Since decision trees can only split along individual feature axes, our model will end up trying to build a staircase to fit a line, which will look something like this:

enter image description here

As you can see this will not generalize perfectly to new data. We can have circles above the true decision line that are under our decision boundary and vice versa for crosses.

However, if we add a+d as a feature then the problem becomes trivial for a decision tree. It can ignore the individual a and d features and solve the problem with a single a+d<0 decision stump.

enter image description here

However, if you were using linear regression, then your model would be perfectly capable of learning $a+d$ without adding an additional feature.

In summary, certain additional features can help depending on the type of model you are using, and you should be careful to consider both the data and the model when engineering features.

Ibraheem Moosa
  • 147
  • 7
Imran
  • 617
  • 4
  • 14
  • 1
    This is exactly the point. Choice of features and choice of model must be considered together. It is a common pitfall to try and reason about feature selection without considering the type of model being used. – Imran Dec 28 '17 at 18:09
  • 1
    For example if you tried the same thing with linear regression then `a` and `d` would suffice and adding `a+d` as a feature wouldn't make a difference. – Imran Dec 28 '17 at 18:18
  • I have updated my answer to make this more explicit. – Imran Dec 28 '17 at 18:21
  • 1
    Furthermore, splitting across the diagonal line requires one split. The staircase you drew "uses up" seven splits. – Acccumulation Dec 28 '17 at 19:52
3

A constructed feature like total can still be predictively useful if it isn't strongly correlated with other features in the same model. total in particular need not be strongly correlated with attack or defense. For example, if attack is (8, 0, 4) and defense is (1, 9, 6), then the correlation of total with attack is 0 and the correlation of total with defense is $\frac{1}{7}$.

Additionally, is that not a problem, for classificators such as kNN, that "total" will be always bigger than "attack" or "defense"? Thus, even after standarization we will have features containing values from different ranges?

If you want to standardize your predictors, you should do it after they've all been constructed.

Kodiologist
  • 19,063
  • 2
  • 36
  • 68
  • 1
    is this really true? Certainly, in a simple linear model, it is not: the matrix `[attack, defense, total]` is of course rank 2. I could imagine in something like a penalized linear model it could make a difference, but that's based on intuition rather than fully working through it. Can you explain why if `attack` and `defense` are not strongly correlated with `total` (which happens when `attack` and `defense` are strongly negatively correlated), why `total` can be helpful? – Cliff AB Dec 30 '17 at 17:41
  • 1
    @CliffAB In hindsight, I was a bit glib here. I was right in saying that a constructed feature can be useful when it's not strongly correlated with other predictors, and that `total` need not be strongly correlated with `attack` or `defense`, but you would never use two predictors and their sum in the same model, because of the linear dependency, with implies a strong correlation between *some* two of the three. – Kodiologist Dec 30 '17 at 17:51
1

To give a general answer, feature engineering in most cases is about extracting meaningful features from your data, so if you give more information to your model, it obviously should behave better. Say that your data consists of e-mail addresses in form ‘name.surname@domain.country-code’. If you used them as-is in your model, each person would be characterized by a unique e-mail, so this wouldn’t tell us much. It would tell us only that one e-mail possibly belongs to different person then another. With feature engineering, from such addresses you could extract information about possible gender (name), family background and ethnicity (surname), nationality (domain) and many more - it gives you pretty much information, doesn’t it?

Tim
  • 108,699
  • 20
  • 212
  • 390
1

What are you trying to accomplish with your "feature" total? If you're merely comparing heroes, attack and defense might be more useful. If you would find the type of build (how offensively-oriented versus how defensively-oriented) to be useful, perhaps attack / defense would be more useful. Or maybe MyAttack - YourDefense is more useful.

It really depends on your goal and it boils down to you injecting additional knowledge into the problem so you can get better answers. You may have heard people throwing around log and squared and ratio and all kinds of ways you could make features, but the bottom line is that "useful" depends on the task at hand and involves transforming the data you have into a domain where decisions are simpler.

Wayne
  • 19,981
  • 4
  • 50
  • 99