5

Assume having several numerical, multidimensional time-series. As preprocessing of further Analysis, I firstly check for relevance and then for redundany of all dimensions/Features.

1) Check for relevance: I will exclude all dimensions with a variance of 0 over the whole dataset - since this specific Dimension does not contain Information that helps to classify/distinguish the time-series from each other.

2) Check for redundancy: I compute the correlation of all dimensions/Features with each other and my Intuition says (here is my question) that those Features-pairs which correlate by either -1 or +1 are redundant. Whereas a high correlation such as 0,99 seems to be redundant, it is not. Only a correlation of either -1 or +1 means redundancy. Therefore i will randomly exclude on the two dimensions/Features which correlate by +1 or -1.

I am yet sceptical whether or not this is a correct assumption. Are there any leads that could prove / disproves my intuiton of the Connection between redundancy and correlation?

Nikolas Rieble
  • 3,131
  • 11
  • 36

2 Answers2

3

High absolute correlation does not imply redundancy of features in the context of classification.

An example is given in the textbook Feature Extraction - Foundations and Applications by I. Guyon et al. (p.10, figure 2 (e)) I reproduced the example for visualization with matplotlib and in Python.

enter image description here

In this example both features correlate highly, yet seperation of classes will only be achieved if both features are used. Therefore correlation does not imply redundancy.

Nikolas Rieble
  • 3,131
  • 11
  • 36
  • 3
    Interesting example, but these variables do not correlate with $r=1$. An example of two observations is $(6,5), (7,5)$. – cangrejo Oct 25 '17 at 10:31
0

It would be helpful to know what kind of analysis you are doing with the data. What questions are you trying to answer? This would help you come up with a better threshold for correlations.

When I do classification work with data, I typically filter correlations higher than 0.85 because high correlation variables are not giving a whole lot of new information for my classification problem. As with any thresholding problem, there is a lot of subjectivity involved. Whatever threshold you pick you need to justify. (This is why knowing what kind of analysis you are doing would be important for choosing a threshold)

Doc Yall
  • 61
  • 4
  • What I want to do is outlier detection. The threshold I consider is 1. I am wondering whether there is a General theory of Information Content and correlation. Does a correlation of 1 ( or -1) really indicate that all Information contained in two time-series could be represented by only one of them?= – Nikolas Rieble Oct 28 '16 at 14:25