8

What methods are there for measuring the strength of arbitrary, highly non-linear relationships between two paired variables? By highly non-linear, I mean relationships that can't sensibly or reliably be modelled by regression to a known model. I'm particularly interested in time-series, but I imagine any thing that works for bi-variate data would work here (if we treat the two time-series as a set of pair data points)

Two that I am aware of are Mean Square Difference (ie. mean square error, treating one time-series as the "expected" value, and one as the observed), as and Distance Covariance. What others are there?

Clarification: I'm basically asking about dependence between series, where linear correlation or simple non-linear correlation (after log, exp, trig, other simple analytic transformations) doesn't really mean much.

naught101
  • 4,973
  • 1
  • 51
  • 85
  • If your focus is on forecasting, you should be aware of the difference between good model fit and predictive ability derived from even a simple multivariate linear model. I posted a question on a related subject here: http://stats.stackexchange.com/questions/25381/relationship-between-r2-and-mae-in-forecasting. – Robert Kubrick Apr 12 '12 at 17:19
  • Non-linear models are a vast area. I suspect you might also be interested in patterns recognition which is a close cousin of non-linear modeling when applied to forecasting. Can you make the question more specific, maybe with an example of your problem? – Robert Kubrick Apr 12 '12 at 17:23
  • I'm not concerned about about forecasting or modelling (except in a very loose sense). Just about measures of dependence between two time-series. As an example, how do I look at the differences between daily rainfall during 1978 in Sydney and Newcastle? – naught101 Apr 13 '12 at 01:08
  • 1
    Apparently there is no simple answer :) http://www.amazon.com/Nonlinear-Series-Analysis-Holger-Kantz/dp/0521529026 – Robert Kubrick Apr 13 '12 at 01:46
  • Hah. I have the book in my hands right now (even the same edition!) :) I've started reading it, but haven't got very far yet. Will report back, perhaps with some answers to this question :D – naught101 Apr 13 '12 at 02:00
  • 1
    It's not totally clear to be what you are trying to measure, but I'll try to give you info that might help. There are correlation measures like Cronback's Alpha that can be used to assess the internal consistency/ relationship among a set of variables. You could also use things like general additive models (GAMs) to test whether the functional estimate is constant. This would imply no relationship between your variables. See the answer here for a discussion on this: https://stats.stackexchange.com/questions/35893/how-do-i-test-a-nonlinear-association – StatsStudent Jun 19 '17 at 19:51
  • 1
    @StatsStudent thanks for the link, it's really helpful. I think it's the best answer so far, if you move it to an answer and no better one appears until the deadline, I'll award you the points. – Allen Wang Jun 23 '17 at 13:49
  • @AllenWang, great. I've moved it to an answer. Glad you found it helpful. – StatsStudent Jun 23 '17 at 23:27

7 Answers7

2

Plain old linear regression has a nice non-parametric interpretation as the average linear trend across all pairs of observations; see Berman 1988, "A theorem of Jacobi and its generalization". So, the data doesn't have to look linear in order to use it; any (broadly) monotonic trend could be summarized this way.

You could also use the Spearman rank correlation... and probably much else besides.

guest
  • 2,381
  • 14
  • 11
  • Thanks, but I'm aware of that, and that's specifically *not* what I was asking for (since a straight line is more or less the simplest model possible, that's implied in my question). I've clarified the question. – naught101 Apr 12 '12 at 05:19
2

The "Amount of relationship" between two discrete variables $X$, $Y$ is formally measured by mutual information : $I(X,Y)$. While the covariance/correlation is somehow the amount of linear relationship, mutual information is somehow the amount of (any kind of) relationship. I'm pasting the picture form Wikipedia's page :

enter image description here

For continuous variables, the information-theoretic concepts are often defined as well but less manageable, maybe less meaningful. I don't want to bother for the moment. Let's stick to discrete variables. Anyway it makes sense approximating continuous variables by discrete ones (using slices) especially in information theoretic approaches.

The problem with information theoretic concepts is often their impracticability. Being able to approximate the mutual information between $X$ and $Y$ is the same as being able to find arbitrary non-linear relationship between them : you need a statistical power (quantity of data) most often far beyond what is reasonable : for any possible value for $x$, you need many (say 1000) samples to compute an estimation of each $P(Y=y|X=x)$. This is not possible in most machine learning or statistical analysis problems. It is kind of logical : if you allow a model to be able to express "any possibility", then it can only be trained by an amount of data covering any possibility several times.

But maybe such an approach is possible, for low dimensional variables, if you enforce low precision : decompose the domains of $X$ and $Y$ into a number of slices small enough so that it is ok for your data. Anyway I think this requires some research.

Benoit Sanchez
  • 7,377
  • 21
  • 43
1

Eventually the most general form of an injective function is

$f(x) = y$

and you can use a discretized version of that function as a model for your data.

Then the problem reduces to determining the expected $y$ for separate regions $a<x<b$.

The method is not powerful because of the high amount of degrees of freedom in the model. Although, that is also inherent to the problem which desires a high degree of freedom (and generality) in the type of functions that can describe the model for the data.

For more specific cases improvements can be made.

Sextus Empiricus
  • 43,080
  • 1
  • 72
  • 161
  • My proposed model was extremely general. You can also use splines, piecewise linear functions, or any of those type of general fitting functions. – Sextus Empiricus Jun 19 '17 at 19:41
1

You may try maximal information coefficient. It outperforms selected methods in the paper and works well in detecting nonlinear relationships between two random variables.

Marks
  • 356
  • 2
  • 6
1

Needs to be a method that is quick to calculate, similar to correlation, but can detect quadratic relationships for example.

The Spearman correlation, which was mentioned in another answer, fits the bill. It is calculated by simply converting the data to ranks and then finding the Pearson correlation for the ranks. It can detect any monotonic association.

There's also the Kendall correlation. The Kendall correlation has a nice interpretation as (a rescaled version of) the probability that ranking cases on one variable will agree with ranking them on another variable. The Spearman correlation, by contrast, is a bit opaque—who thinks about data in terms of linear relationships between ranks? The Kendall correlation is not "quick to calculate" in terms of computational complexity (it's $O(n \log n)$ whereas Spearman is $O(n)$), but it requires no human judgment to compute and it's already implemented in a lot of statistics software, and with a modern machine, the asymptomatic complexity is unlikely to matter except with the very largest datasets.

Kodiologist
  • 19,063
  • 2
  • 36
  • 68
  • If a [comparison sort](https://en.m.wikipedia.org/wiki/Comparison_sort) is used to compute ranks, then Spearman will also be $n\log(n)$. – GeoMatt22 Jun 23 '17 at 23:50
  • @GeoMatt22 Ah, it looks like the cs.stackexchange answer I linked to didn't take the ranking step into account. So the Spearman correlation is probably no faster than the Kendall correlation, after all. – Kodiologist Jun 24 '17 at 00:28
1

It's not totally clear to be what you are trying to measure, but I'll try to give you info that might help. There are correlation measures like Cronback's Alpha that can be used to assess the internal consistency/ relationship among a set of variables. You could also use things like general additive models (GAMs) to test whether the functional estimate is constant. This would imply no relationship between your variables. See the answer here for a discussion on this: How do I test a nonlinear association?

StatsStudent
  • 10,205
  • 4
  • 37
  • 68
0

I cannot comment thus I have to post the answer. Have a look at Dynamic Time Warping, simple algorithm that can kind of detect/compare patterns between two time series, which can have even different granularity. https://en.wikipedia.org/wiki/Dynamic_time_warping

reicja
  • 63
  • 1
  • 7