4

I currently have a model matrix $X$ with $6$ columns, which is being used for a factorial design problem, with each column associated with an effect. The ultimate goal is to be able to estimate as many of the effects, $A,B,C,D,E,F$, as we can. This is to be done by using the least squares equation: $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$, where $Y$ is a vector of responses which I didn't include as it is not need here.

$$X = \begin{bmatrix} A & B & C & D & E & F \\ \hline -1 & 0 & -1 & -1 & 0 & -1 \\ -1 & 0 & -1 & 1 & 1 & 0 \\ -1 & 0 & 1 & -1 & 0 & 1 \\ -1 & 0 & 1 & 1 & -1 & 0 \\ 1 & -1 & 0 & -1 & 0 & 1 \\ 1 & -1 & 0 & 1 & -1 & 0 \\ 1 & 1 & 0 & -1 & 0 & -1 \\ 1 & 1 & 0 & 1 & 1 & 0 \\ \end{bmatrix} $$

My correlation structure, $Corr(X)$, where the correlation is computed as the correlation between the 6 columns of $X$, looks like:

$$Corr(X) = \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 & 0.5 & -0.5 \\ 0.0 & 0.0 & 1.0 & 0.0 & -0.5 & 0.5 \\ 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 \\ 0.0 & 0.5 & -0.5 & 0.0 & 1.0 & 0.0 \\ 0.0 & -0.5 & 0.5 & 0.0 & 0.0 & 1.0 \\ \end{bmatrix} $$

My question is, how many of the $6$ original effects can be estimated and how do I go about determining this? I saw in a paper that looking at the correlation matrix, we can see that five of the six effects can be estimated, and this is because $dim(space) = 5$, or the dimension of the correlation matrix spanned by six columns is $5$.

I am not exactly sure why this is true. Could anyone shed some insight how one can determined which effects may be estimated just by looking at the correlation matrix? I know in general that for OLS to work, the columns of the regressors $X$ must be linearly independent. However, what does looking at the linear independence of a correlation matrix have anything to do with this? Thanks!!

user1398057
  • 2,175
  • 1
  • 18
  • 24
  • 1
    A closely related question concerns what can be done when all you have is the covariance matrix: see http://stats.stackexchange.com/questions/107597. – whuber Feb 02 '16 at 00:21

1 Answers1

3

The rank of the $X$ matrix is 5. Do not focus on the correlation matrix; it is a red herring at this point. As you correctly assess having a rank of 5 mean that despite $X$ having 6 columns it spans a 5-dimensional space. That is because one of the $X$ matrix's columns in a linear combination of your previous five. For the matrix $X$ presented this is the 6th column. The sixth column can be written as the sum of the third and fifth column minus the second column. (So $F = C + E - B$.) This clearly violates the linear independence condition between the columns of $X$.

Per the OP's request: The heuristic I used to determine which column can be expressed as the linear combination of others is as follows: (Note this is a heuristic not a rigorous methodology):

Given that the rank is 5, instead of 6, we know we look for a single "problematic" column. As such if I took a column out of $X$ (call that thinner $X$, $X^5$) and got the rank of it to be 5 the column removed could be expressed as the linear combination of other columns. I tried that and I immediately got that column $F$ is not independent of the others. Let me stress that there definitely not a single column you can remove. That is because if $F = C + E - B \Leftrightarrow B = C + E - F \Leftrightarrow C = \dots $, etc. etc. So OK, I found that $F$ is a problematic column, how I get what linear combination of the other columns reconstructs it? To do this, I solve the corresponding linear system $X^5$ over $F$. I used MATLAB so I literally typed X(:,1:5)\ X(:,6) but in R I would have typed qr.solve(X[:,1:5], X[:,6]) to get the solution as 0.0000, -1.0000, 1.0000, 0.0000, 1.0000. So you have it $F = 0A -1B + 1C + 0D + 1E \Rightarrow F = C + E - B $. Easy right?

usεr11852
  • 33,608
  • 2
  • 75
  • 117
  • Side comment: The correlation matrix reported appears wrong. For example the correlations between columns $B$ and $C$ has to be zero. There is not correlation whatsoever while in the matrix provided they are anti-correlated ($\rho = -0.5$). – usεr11852 Feb 01 '16 at 19:49
  • Thanks, I fixed the correlation matrix! Just a quick question, do you have any idea why the paper focused on the correlation matrix? Can you determine linear dependency by looking at the dimension of the correlation matrix as well? Also, do you mind if I asked how you were able to so quickly determine that the 6th column is a sum of third and fifth minus the second? thanks!!! – user1398057 Feb 01 '16 at 20:13
  • I am glad I could help. You do not mention any papers in your question so I cannot comment on that. What I said for $X$ applies for the correlation matrix too. Eg. take the 4th row: $0 + 0 - 0 = 0$ or the 5th row: $-0.5 + 1 -0.5 = 0$, etc. etc. I will describe the *heuristic* I used in the question. – usεr11852 Feb 01 '16 at 21:30
  • Thanks so much, when you write "eg. take the 4th row: $0+0-0=0$ or 5th row" I am not quite sure how you get the two equations and how it relates to why I can do the same thing with a correlation matrix. This might be a dumb question but do you mind if I ask what these equations are? – user1398057 Feb 01 '16 at 22:01
  • I refer at the $i$-th row of the matrix you denote $Corr(X)$. So let's say you look at the second row: $C_{col} = 0$, $E_{col} = 0.5$ and $B_{col} = 1$ so you have $0 + 0.5 -1 = -0.5$ which actually your value in $F_{col}$. In the case of $i =1 \& 4$ everything equates to zero so the equality is trivial. For for $i=5$ $B_{col} = 0.5$, $C_{col} = -0.5$ and $E_{col} = 1$. So you have $ -0.5 +1 -0.5 = 0$ which again equates to the value you have in column $F_{col}$ at row $i=5$. In general as mentioned the correlation matrix is a bit unnecessary here all the information need can be found in $X$. – usεr11852 Feb 01 '16 at 22:10
  • I think I see what I was confused about, in the above matrix $X$, it was a linear combinations of entire column vectors, but in the correlation matrix, it was done row by row to keep with the notation, but ultimately it is the same, but I failed to see immediately. Thanks again, you've greatly helped me clear a lot of confusion! – user1398057 Feb 01 '16 at 22:23