25

What is the definition of "feature space"?

For example, When reading about SVMs, I read about "mapping to feature space". When reading about CART, I read about "partitioning to feature space".

I understand what's going on, especially for CART, but I think that there is some definition that I have missed.

Is there a general definition of "feature space"?

Is there a definition that will give me more insight into SVM kernels and/or CART?

power
  • 1,564
  • 1
  • 16
  • 29
  • 8
    Feature space just refers to the collections of features that are used to characterize your data. For example, if your data is about people, your feature space might be (Gender, Height, Weight, Age). In a SVM, we might want to consider a different set of characteristics to describe the data, such as (Gender, Height, Weight, Age^2, Height / Weight) etc; this is the mapping to another feature space – JCWong Dec 22 '12 at 17:14
  • Would you please give the names/titles of you reading? – fu DL Aug 07 '19 at 12:40

1 Answers1

49

Feature Space

Feature space refers to the $n$-dimensions where your variables live (not including a target variable, if it is present). The term is used often in ML literature because a task in ML is feature extraction, hence we view all variables as features. For example, consider the data set with:

Target

  1. $Y \equiv$ Thickness of car tires after some testing period

Variables

  1. $X_1 \equiv$ distance travelled in test
  2. $X_2 \equiv$ time duration of test
  3. $X_3 \equiv$ amount of chemical $C$ in tires

The feature space is $\mathbf{R}^3$, or more accurately, the positive quadrant in $\mathbf{R}^3$ as all the $X$ variables can only be positive quantities. Domain knowledge about tires might suggest that the speed the vehicle was moving at is important, hence we generate another variable, $X_4$ (this is the feature extraction part):

  • $X_4 =\frac{X_1}{X_2} \equiv$ the speed of the vehicle during testing.

This extends our old feature space into a new one, the positive part of $\mathbf{R}^4$.

Mappings

Furthermore, a mapping in our example is a function, $\phi$, from $\mathbf{R}^3$ to $\mathbf{R}^4$:

$$\phi(x_1,x_2,x_3) = (x_1, x_2, x_3, \frac{x_1}{x_2} )$$

soufanom
  • 1,215
  • 1
  • 9
  • 16
Cam.Davidson.Pilon
  • 11,476
  • 5
  • 47
  • 75
  • How does this differ from a sample space in probability theory? Just asking. I would like to know. – Placidia Dec 23 '12 at 02:43
  • It's is very similar, if not identical. If you consider the data-generating distribution $D$, then the feature-space is identical to the support of $D$. – Cam.Davidson.Pilon Dec 23 '12 at 04:25
  • 1
    I would say that, as Pilon's example shows, feature space can be increased by extracting some new features. Sample space in probability can't. It's exhaustive, feature spaces aren't. – QuestionEverything Apr 05 '16 at 12:56
  • @Cam.Davidson.Pilon someone wsa inspired by your answer it seems: https://dataorigami.net/blogs/napkin-folding/17536555-feature-space-in-machine-learning – AIM_BLB Aug 06 '19 at 15:22
  • 1
    @AIM_BLB that's me! – Cam.Davidson.Pilon Aug 06 '19 at 15:44
  • Ahaha, in that case great post x2 :) – AIM_BLB Aug 06 '19 at 15:46
  • @HasanIqbal What does "It's exhaustive" mean? Take the iris dataset as example, there are 150 instances and 4 attributes? Would you please illustrate "It's exhaustive, feature spaces aren't"? – fu DL Aug 07 '19 at 12:53
  • @HasanIqbal What do you think about the probability distribution on $\mathbb{R}^4$ induced by function $(x_1, x_2, x_3) \mapsto (x_1, x_2,x_3, x_1 / x_2)$? Could that be considered effectively the same as enlarging the sample space? – Quinn Culver Nov 03 '19 at 05:06