I've tried reading many explanations of CCA, and I don't understand it. For example, on Wikipedia, it refers to two "vectors" $a$ and $b$ such that $\rho = \text{corr}(a^{\top} X, b^{\top} Y)$ is maximal. But if $a$ and $X$ are vectors, isn't $a^{\top} X$ a scalar? What does it mean for two scalars to be correlated?
Other explanations use matrix notation without any dimensions, e.g.
Can someone explain CCA with a small example and all the dimensions provided?
In a uni/multivariate regression, one tries to predict observations (Y) based on a linear combination of predictor variables (Xb). This assumes a linear relationship between the predictor and observed variables.
Y = Xb
In this situation, X can be of any dimensions i.e. can contain any number of predictor variables in its columns, whereas Y is a vector of observations. The correlation value indicates how good the model in X explains observed data in Y. This is also referred as explained variance. b is a vector that maximizes the explained variance.
Canonical correlation asks a similar question in a more general context. It is most useful when both predictor (X) as well as observed data (Y) results from multiple data sources i.e. they are both matrices.
Assume we collected some behavioral responses in an experiment. These responses can be described with 3 different characteristics such as speed, reliability, amplitude, etc. When we do this for N different individuals we would end up with a matrix of 3 by N. Assume now that for each of these N individuals, we have also 10 psychological variables (IQ, anxiety, impulsiveness, etc.) that we collected before the experiment, leading to 10 by N matrix. Note that both Y and X have the same number of observations, but may have different number of data sources. Now we can ask the question of how to understand the relationships between these variables. More precisely, how does psychological variables can be used to predict behavioral parameters? For example, does the data supports the hypothesis that a combination of anxiety and impulsiveness jointly predicts a faster and smaller behavioral responses?
Cannonical Correlation finds the best combination of both datasets in Y and X that maximizes their correlation. It creates a new set of variables, called canonical variables, U and V by linearly combining observations. The number of canonical variables depends on the number of data sources in X and Y. In the previous example, there will be 3 different canonical variables, which is the min(rank(X),rank(Y)).
U = Xa
V = Yb
The first column of U and V contains the first canonical variables. The key properties of U and V is that the correlation between the first canonical variables is the highest correlation you can achieve with this dataset when characterizing linear relationships. The second and third canonical values will be the second and third highest correlation values, respectively.
Canonical correlation can therefore be useful to understand how different set variables can be related to each other.
