0

I have two groups of animals and 60 to 100 variables per animal (describing their behavior during 24h). I want to know whether the two groups of animals behave differently (I have 11 data points per group), and I do not care much about knowing what is different.

I started by doing a SVM followed by a permutation to assess the accuracy of the SVM statistically. Got a good accuracy (5/6 correct prediction) but it is far from getting to the significant threshold (p>0,2), due to the over-dispertion of the distribution of accuracy scores, probably linked to the the low sample size.

Now I am wondering about a most simpler approach: making a PCA and do a basic wilcox test on the first component (keeping p<0.05 as the threshold, since I am doing only one test). I think it is allowed: since the PCA does not use the grouping variable, it will treat inter- and between- groups variance similarly. The fact that the p-value with 11 animals per group is super super low make me wonder if I do not make any wrong assumption here.

Ferdi
  • 4,882
  • 7
  • 42
  • 62
Julien Colomb
  • 113
  • 5
  • 1
    What you're calling a Wilcox test is presumably Wilcoxon-Mann-Whitney. At a guess you used R: the truncated name there presumably arose from a mistaken preference for brevity -- or perhaps because some prankster wanted to sprinkle minor confusion in the literature. – Nick Cox Nov 24 '17 at 16:48
  • sorry, you guessed right for R, I edited the title. – Julien Colomb Nov 27 '17 at 11:36

1 Answers1

2

  1. Your PCA technique might miss the difference between the groups: That is because it focuses on the total variation rather than the variation between the groups.

    If you are focused on exploring/picking-out a difference between the groups, then a linear discriminant analysis (LDA) might be a better (stronger) tool.

    But since you started with SVM (support vector networks?) instead of MANOVA/LDA I wonder if you have some restrictions that made you choose a non-linear technique? PCA is not helping on such issues that much.

    See the image below for a simple example that expresses these effects. The largest variations in the variables between groups (e.g. a total of var1 + var2) might be perpendicular to the differences within the groups (e.g. a distribution among var1 and var2). demonstarting multivariate differences

  2. Aside from this limitation in power your PCA technique is robust regarding type I errors. Given the null hypothesis that 'the two groups are the same', then you could sample from a single distribution, perform PCA, and assign groups afterwards. So the PCA can't have any effect, if the null hypothesis is true.

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Sextus Empiricus
  • 43,080
  • 1
  • 72
  • 161
  • While I said that your PCA+WMW technique is 'robust regarding type I errors', the fact that you already did some other analysis, and now are looking for something different (probably because the previous result did not make you happy), makes your approach as a whole *not* robust (the approach to try multiple things until finding something significant makes it more likely to find something significant). At best you can analyze if something went wrong with the SVM. Also, given the small sample, you can report differences, while not caring about significance (let the next experiment do that). – Sextus Empiricus Nov 24 '17 at 16:56
  • Thanks, That is what I thought, but the results made me doubt myself. Since I have many variables for few animals, I was afraid a manova would be too prompt in type 2 errors and I indeed started with svm hoping to get a technique that would not miss existing difference with less false positive. I was very surprised to see that the SVM could not make significant prediction, while the PCA first discrimant was so much different in the 2 groups. I think this is due to a big difference in the spread of the data, but a large overlap, which becomes overrepresented using prediction on 6 animals. – Julien Colomb Nov 27 '17 at 11:32
  • And thank you for the reminder about robustness, I am working with a positive control now (young versus old animals), trying to find the best strategy to apply to future experiments. So I am safe there :) It is behavior, so some non-linearity is to be expected, but in real, we just do not know. – Julien Colomb Nov 27 '17 at 11:35