6

"Large $p$, small $n$" typically refers to "many independent variables, few samples".

In my case, I have $1$ independent variable, $300$ dependent variables, and $n < 20$ samples.

Thus, my case is not the typical "large $p$, small $n$" scenario (even though I do have many regression coefficients to estimate -- it's just that they are not due to the independent variables, but due to the number of dependent variables).

How is my situation classified? Does it fit into the "large $p$, small $n$" scenario, albeit I have actually less independent variables than samples?

Note: I would like to do a multivariate regression or MANOVA on my data, but have a "multicollinearity" problem. But it is not really multicollinearity because I only have one independent variable. What's going on here?

amoeba
  • 93,463
  • 28
  • 275
  • 317
cmo
  • 324
  • 2
  • 12
  • How do you know you have a multicollinearity problem? Do you have high correlation in your dependent variables? Also, you sure as heck can have multicollinearity in your dependent variables. "Multicollinearity" just means "at least one variable is (approximately) linearly dependent on another variable." – shadowtalker Sep 24 '14 at 19:34
  • @ssdecontrol I know I have a multicollinearity problem because trying to fit a multivariate regression or do a MANOVA test fails due to multicollinearity/singularity (says Matlab) – cmo Sep 24 '14 at 22:40
  • 1
    Can you clarify your situation? Do you simply want to test if there is change across your IV? Do you want to predict values for all DVs for future observations? Are you trying to run Fisher's linear discriminant analysis? – gung - Reinstate Monica Oct 31 '14 at 15:31

2 Answers2

5

There are several combinations on the size of $n$ and $p$: small $n$ - large $p$, small $p$ - large $n$, large $n$ - large $p$ ... See Johnstone & Titterington, 2009, Statistical challenges of high-dimensional data for an overview.

In your case, it appears that you have a small $p=1$ and relatively small $n$, with high dimension of the dependent variable. It is likely that your independent variable may not contain enough information to properly model $300$ responses.

The justification of this claim is as follows. If you use a GLM for your data, then you have $20$ samples to estimate $300+$ parameters in the covariance matrix of the errors. This may induce over-fitting, and the precision of the estimators will be unnecessarily vague (in the sense that confidence intervals for these parameters might be too wide) and inaccurate (far from the true value). However, if you restrict the structure of the covariance matrix, then it may be possible to estimate the parameters more accurately (How to restrict the structure of the covariance matrix? That's a big question which depends on the context). Moreover, the fewer covariates you use, the more "responsibility" the residual errors carry to explain the unobserved variability. This may, for instance, inflate the variances or induce the need for more flexible distributions than normal for modelling the residual errors.

Additional references of possible interest:

Community
  • 1
East
  • 109
  • 2
  • 3
    (+1) Welcome to our site! Your mastery of the $\TeX$ markup, clear restatement of the question, pithy answer, and ability to find and cite references show you are already expert at contributing to forums like this. I hope you will continue to participate! – whuber Sep 24 '14 at 20:05
  • How is the model of this type? The question says that the response ${\bf y}$ is of dimension $n \times 300$ while there is only $1$ predictor. – Juho Kokkala Sep 24 '14 at 20:14
  • @JuhoKokkala You just made me doubt about my reading skills ... I think you are right, and I have modified my answer. – East Sep 24 '14 at 20:24
  • 1
    (I misread the question, too.) Why, though, do you suggest there might not be enough information? There's enough data to estimate the entire covariance matrix, after all. I admit the estimates will be pretty uncertain, but no individual component of that matrix will be any more uncertain than if the problem involved only two dependent variables. Would you continue to maintain there would not even be enough information here if $300$ were changed to $2$? Sure, there are multiple testing problems, but the situation is better than if one had to perform $300$ separate ordinary regressions. – whuber Sep 24 '14 at 20:31
  • @whuber I have added some discussion on your points. Thanks for the feedback. – East Sep 24 '14 at 20:59
  • Thanks. I am not sure what exactly you mean by "inflate the variances": that phrase has an ambiguous referent. Regardless, what's holding up my understanding is the thought that one could just run 300 separate regressions. By giving up all information about correlations among the dependent variables, you would lose information about correlations among the 600 parameter estimates: but it seems the residuals shouldn't be drastically affected by that procedure. If anything, they ought to be slightly reduced overall, thereby *decreasing* estimated residual variances and parameter standard errors. – whuber Sep 24 '14 at 21:09
  • @whuber But if you run 300 separate regressions, you will end up with 300 estimators of the (single) regression coefficient.The claim about the variances is rather empirical. For example, if the sample contain outliers, the estimation of the variance is typically "inflated" by these. – East Sep 24 '14 at 21:11
  • 2
    Aha! This important assumption appears to be missing from the question--or maybe it is understood but not stated? I had taken the model to be of the form $y_i = \alpha_i + \beta_i x + \varepsilon_i$ for each dependent variable as indexed by $i$, requiring estimating $300$ parameters $\{\alpha_i, \beta_i\}$. An example would be a collection of about $20$ time series (such as daily measurements of $20$ experimental subjects) with a covariate $x$ available with each time series (such as a characteristic of each subject). – whuber Sep 24 '14 at 21:18
  • (-1) After the misunderstanding was cleared, I feel that this answer does not address the actual question that was asked. As I read it, the question is primarily about terminology. – amoeba Oct 31 '14 at 16:03
1

[As I read it, the question is primarily about terminology, and @East's answer (good as it is) does not explicitly address it.]

Sometimes the distinction between dependent and independent variables is not so clear. As you are referring to MANOVA, you probably have $300$ variables measured for two groups. Technically, you are right, it is $300$ dependent variables, but imagine that you want to predict the group membership by looking at the variables (after all, the purpose of running MANOVA is to test if the groups are different or not). Now group identity suddenly becomes a dependent variable, and you have $300$ independent variables to make the prediction.

So I think the distinction between dependent and independent variables is not very important here, and your situation can be safely described as "large $p$, small $n$".

In practice, people definitely do refer to classification problems, e.g. linear discriminant analysis, with number of features $p\gg n$ as "large $p$, small $n$" (see e.g. The Elements of Statistical Learning 18.2). But linear discriminant analysis is almost the same thing as MANOVA, see here: How is MANOVA related to LDA? So I would advocate to go ahead and to call it "large $p$, small $n$" in MANOVA context as well.

Community
  • 1
amoeba
  • 93,463
  • 28
  • 275
  • 317