1

The main purpose of my work is to discriminate patients vs healthy controls using fMRI and multivariate pattern analysis (MVPA). Since I want to classify at the subject level I performed a separate subject GLM in order to get the parameter estimates for each task condition. Then I transformed beta estimates into t values in order to improve signal to noise ratio. I have 31 controls and 32 patients. All of them completed 3 runs except for 4 patients that only completed 2 runs due to fatigue.

My question is: since MVPA is very sensitive to uneven classes should I exclude the 4 patients that only completed 2 runs and then balance classes by doing some kind of undersampling of the controls class for instance? My main concern is that those 4 patients have lower signal to noise ratio when compared to the others that completed 3 runs which could undermine classification accuracy.

I searched through several papers and some textbooks but I cant find anything on uneven runs in between-subject classification analysis. I would appreciate it if someone shared some references about this topic.

user234582
  • 13
  • 2
  • Could you please say more about the details of your model? It’s possible that a mixed-model GLM could by itself deal with this type of imbalance, which is one of the advantages of that type of approach. It’s hard to tell from your description just how you have done your analyses, however. You will have to provide more details about the data and your model to get a useful informed answer. – EdM Jan 31 '19 at 03:37
  • Thanks for your reply EdM! Abou my fmri experiment: 31 controls and 32 patients performed 3 runs of a visual task except for 4 patients that only completed 2 runs; then I used a separate general linear model in each subject which yielded beta estimates for the task-related condition for each voxel; this beta estimates were transformed to t values by computing a t test against 0 for each voxel; in sum, I have one t-map per subject that I entered as input to a nested cross-validation classification analysis. – user234582 Jan 31 '19 at 13:08
  • I am using brainvoyager to perform the GLM and libsvm to do the classification. In a separate subject GLM, I think brainvoyager gets beta estimates for all the runs that a particular subject has performed and then takes the average of all of them to yield a single value of beta per stimulus per subject. The problem here is that classifiers are usually sensitive to small imbalances in classes and in this case I have 4 patients (4 elements of one class) that have beta estimates resulting of only one run which differs from the other elements (and most importantly from the other class). – user234582 Jan 31 '19 at 13:16
  • I suppose the mixed-model GLM that you suggested corresponds to the random effects general linear model (RFX-GLM) as it is generally reported in fMRI literature. However, this RFX-GLM is the standard approach to univariate fMRI analysis. In contrast, the analysis I am trying to implement is the classification of patients vs controls using several voxels simultaneously (multivariate pattern analysis). Since classification analysis rely on independence of examples (in this case subjects) I think it is not valid to do RFX-GLM. – user234582 Jan 31 '19 at 13:31

1 Answers1

1

You really don't have a imbalance problem per se, with 31 controls and 32 patients. What you have is different precision of the within-subject errors, as 4 patients only completed 2 out of 3 runs (presumably for reasons not related to the imaging results per se). The simplest solution would be to down-weight the cases having only 2 instead of 3 runs, corresponding to the lower precision of the per-voxel estimates in those cases. This would presumably be something on the order of $\sqrt{2/3}$ relative to the other cases.

This is relatively straightforward to handle with penalized logistic regression (ridge or LASSO), which has a very close connection to SVM as explained in Section 9.5 of ISLR. Standard statistical software for logistic regression allows for such case weighting. I do not have experience with libsvm, but if that also allows for case-specific weights then that would provide an answer.

Also, consider what the authors of ISLR have to say about the relative performance of logistic regression and SVM (page 357):

When the classes are well separated, SVMs tend to behave better than logistic regression; in more overlapping regimes, logistic regression is often preferred.

So logistic regression might be preferred unless there are very clear imaging distinctions between your control and patient classes.

You have a fairly difficult problem here, with only 31 cases in the smaller group and a desire to use multiple voxels as features for classification. To avoid overfitting you are limited to about 2 or 3 unpenalized features, or a larger number of penalized features, with that sample size. (The rule of thumb is 10-20 cases in the smaller group per unpenalized feature.) Tuning parameters help prevent overfitting in SVM, similarly to the way that penalized logistic regression works; Chapter 9 of ISLR explains this issue in more detail. My sense is that proper handling of penalization to avoid overfitting will be a more pressing problem here than the relatively small difference in precision due to having only 2 instead of 3 observations for some subjects.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • Thank you so much EdM! I wasnt aware of this case weighting possibility but this might really be the solution to my problem. I looked through libsvm and it actually allows instance weighting. I will also check logistic regression as well. Could you just explain the rationale for setting the weights specifically to square root (2/3)? – user234582 Feb 01 '19 at 00:03
  • @user234582 the [standard error of an estimate of a mean value](https://en.wikipedia.org/wiki/Standard_error) is the standard deviation of the observations divided by the square root of the number of observations; precision scales with the square root of the number of observations. So the precision of estimates based on 2 observations will be only about $\sqrt{2/3}$ of estimates based on 3 observations, if the per-observation standard deviations are the same. If you have better estimates of relative precision, you could use those. – EdM Feb 01 '19 at 00:17