0

  1. I have three time series A, B, and C with the same length.

  2. Calculate pairwise measurements among A, B, and C, and perform pairwise permutation tests among A, B, and C for the correlation with the same number of surrogates N for each test. That gives p-value for the measurement for each pair.

  3. Declare that I have more confidence in the measurement between A and B than in the measurement between B and C if the p-value for A and B is smaller than the p-value for B and C.

Is the last step ok?

hamster on wheels
  • 103
  • 6

1 Answers1

4

The short answer is "No".

Basically you need to remember that p-values are random variables themselves and more importantly remember that the "difference between significant and non-significant is often not significant". Basically it is not hard to find a case where effect A is not significantly different from 0, but effect B is significantly different from 0 (at some specified significance level), but A and B are not significantly different from each other. The same will be true of correlations like you are asking about.

A better approach may be to jointly model all 3 variables together, probably using a Bayesian model, then look at how the joint distribution of the correlations of interest looks.

Alexis
  • 26,219
  • 5
  • 78
  • 131
Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • Thanks for the answer though. I have waited for a long time and no one even made a single comment. – hamster on wheels Aug 04 '17 at 21:00
  • 2
    @rxu waiting for a few *hours* is not really "a long time" on CV, where (good) answers or comments may pop up only after days, weeks, months, or even years. – Alexis Aug 04 '17 at 22:11
  • I got it. you are right in the sense that if measurement X is significant and measurement Y is insignificant, it does not mean that measurement X - Y is significant. Here I am directly comparing the degree of significance for measurement between A and B with the degree of significance of measurement between B and C. The sampling size for both measurements are the same and I am not doing a subtraction here, so it will be fine, I guess. – hamster on wheels Aug 04 '17 at 23:43
  • Thanks for writing this. I realized if I use p-values as my new measurement, it purely just reflect the confidence in rejecting the null hypothesis. – hamster on wheels Aug 05 '17 at 00:25