0

I am given the LSE of $X\beta$, but I need to find the rank of $X$. Is there any result or theorem that allows me to obtain $r(X)$?

Oski
  • 15
  • 4
  • 1
    What do you that you are given the LSE? Without any additional info then no, we can't say anything about the rank except that it's $k+1$, where $k$ is the number of independent variables in the model. – Repmat Oct 04 '21 at 17:22
  • 1
    @Repmat You omitted the crucial "less than or equal to" before "$k+1.$" With the information provided, the best one can say is that $r(X)$ is an integer between zero and the number of columns of $X.$ In other words, the LSE ("least squares error," I presume) provides no information whatsoever about the rank. The reason should be obvious: the LSE depends on the response in addition to $X$ and, given any $X,$ one can adjust the response to reproduce any LSE whatsoever. – whuber Oct 04 '21 at 17:43
  • @whuber That's what I was thinking. I'm thinking that my square matrix $X$ has full-rank, so then the LSE of $\sigma^2$ ($Y'(I-P)Y/(n-r)$) would not exist, since $n=r$. This just doesn't seem right, so I figured that $X$ is not full-rank. – Oski Oct 04 '21 at 18:00
  • 2
    Could you explain what you really mean by "LSE"? Given a response $y,$ it clearly exists for any given $\beta$ as the sum of squares of $y-X\beta.$ If instead you estimate $\beta$ using least squares, this is a globally convex problem and therefore at least one solution exists; it has an LSE; and all solutions have the same LSE. Again, the LSE exists. How, then, do you contemplate it possibly *not* existing?? – whuber Oct 04 '21 at 18:51
  • 1
    Why do you have this question? It seems a bit odd situation to me to derive the rank of the matrix based on LSE. Why and when would that be necessary? – Sextus Empiricus Oct 04 '21 at 19:23
  • 1
    With a full rank matrix the estimate equals the measurement $\hat y = y$ and the residuals are zero. Then the estimate for the variance of the sample data $y$ can not be made and neither the estimate for the sample variance, and error, of the estimated coefficients $\hat\beta$.... but what is the problem? Do you have a full rank matrix and you still got some estimate for the LSE? What is LSE? The sum of squares residuals, or the estimate of the error? The sum of squares residuals do still exists and will be zero. – Sextus Empiricus Oct 04 '21 at 19:34
  • 1
    @whuber generally for a LS estimator you would assume (and I suppose require) full rank of $X$ no? – Repmat Oct 04 '21 at 19:52
  • @SextusEmpiricus The main reason I posted the original question was because I want to find the LSE for $\sigma^2$ based on the least squares estimate for $X\beta$, which requires knowing $r=r(X)$. – Oski Oct 04 '21 at 20:11
  • @whuber I am using "LSE" to mean least squares estimate. So, I am given $X\hat{\beta}_{LSE}$. My ultimate goal is to obtain the LSE for $\sigma^2$, but this requires knowing $r=r(X)$. – Oski Oct 04 '21 at 20:13
  • 2
    Why do you only know $X\hat\beta$ without the number of coeffients? – Sextus Empiricus Oct 04 '21 at 20:17
  • 1
    @Repmat That would be too restrictive, because it frequently is the case that people assemble a set of explanatory variables that are linearly redundant (of reduced rank). Good software automatically handles this by dropping the least number of columns of $X$ needed to make it of full rank. Oski: your phrase "Least Squares Estimate of $X\beta$" presumably is $X\hat\beta$ where $\hat\beta$ is the LSE of $\beta.$ This is usually known as the "fitted" or "predicted" value. It is useless without information related to the actual observations of the response variable. – whuber Oct 05 '21 at 14:05

1 Answers1

1

Without any more information or assumptions, the least squares estimate $\hat y$ tells you nothing about the number of coefficients in $\beta$ or rank of $X$ from which this is computed ($\hat y= X\hat\beta$).

It could have been a single column vector. E.g. say your least squares estimate was $$ \hat{y} = \begin{bmatrix} 1\\4\\3\\7\\12 \end {bmatrix}$$ then maybe you had $$ X = \begin{bmatrix} 2\\8\\6\\14\\24 \end {bmatrix} \quad \text{and} \quad \hat\beta = 0.5$$ or you had $$ X = \begin{bmatrix} 1&0\\1&3\\1&2\\1&6\\1&11 \end {bmatrix} \quad \text{and} \quad \hat\beta = \lbrace 1,1 \rbrace$$ or maybe something else. There is no way to tell and there is more information necessary to say anything about it.

Sextus Empiricus
  • 43,080
  • 1
  • 72
  • 161
  • 1
    Maybe, one possibility to reverse engineer the rank is when you know that the columns should be some sort of know set of functions. E.g. maybe it is fitting with a polynomial? Then you could find out which set was used to obtain $\hat y$. (But this is a big maybe, because we have no idea about your original problem) – Sextus Empiricus Oct 04 '21 at 20:32
  • Thank you for your answer! So, suppose I know that $\hat{Y}=MY$, where $M$ is an $n\times n$ matrix. Would this additional information provide further insight? – Oski Oct 04 '21 at 22:41
  • 1
    @Oski an n by n matrix is potentially full rank, but do you know something more practical about it? In principle you can have some columns repeated or lower rank for some other reasons. – Sextus Empiricus Oct 05 '21 at 05:43
  • 1
    One thing you can deduce is if $\hat{y}\neq y $ (if the LSE is not equal to the observation) then the n by n matrix can not be full rank (because this contradicts that a full rank matrix should lead to a perfect fit). – Sextus Empiricus Oct 05 '21 at 06:12
  • Okay, that's helpful! In my case, $\hat{Y}=MY\ne Y$. So, that means absolutely that $M$ is not full-rank? Is there a source for this? Also, I'm assuming that all we can do is determine that $M$ is not full-rank; we cannot find the rank of $M$ from here still? – Oski Oct 06 '21 at 13:11
  • 1
    Indeed you can find nothing further. Only the full rank gives you necessarily a perfect fit (but perfect fit does not tell you it's full rank because non-full rank matrices might be able to fit perfectly as well and depending on the noise distribution it can do so with non-zero probability). For other lower rank matrices there is no way to make a distinction only based on $\hat Y$ and $Y$, and more information must be used. – Sextus Empiricus Oct 06 '21 at 14:44
  • Alright, thank you so much for your help. I really appreciate it :) – Oski Oct 06 '21 at 14:46
  • 1
    The space for the residuals is the orthogonal complement of the space spanned by X. Full rank matrices have an orthogonal complement of dimension zero. Thus they result in a perfect fit (if you do OLS). https://stats.stackexchange.com/questions/515874/why-are-the-residuals-in-mathbbrn-p – Sextus Empiricus Oct 06 '21 at 14:50
  • 1
    If you would have multiple cases of $Y$ and $\hat Y$ from the same matrix $X$ then you could deduce the dimensionality of the space of $\hat Y$ or the residuals $Y - \hat Y$ and this gives you the rank of $X$. – Sextus Empiricus Oct 06 '21 at 14:51