5

Suppose I have an observation matrix of size $N \times M$ where $N$ is the number of samples and $M$ is the number of variables. If the rank of the observation matrix is $R<M$, does it tell anything useful for the reader, when I am intending to apply machine learning?

The rank is the number of linearly independent variables, if I have understood correctly. So, could I write that the observation matrix is a kind of "good" one if $R$ is not very different from $M$, in the sense that it does not contain (linearly) redundant information? I have $M=150$ and $R=130$.

Edit: If I should report the rank, should I also mention how many principal components explain some number of variation (e.g. first $100$ PCs explain $99\%$ of variation).

sup
  • 51
  • 2
  • 1
    Maybe it could work as an argument for why feature selection is needed later on? Interesting question, I hope this gets great answers. – mmh May 15 '15 at 08:07

1 Answers1

1

Wikipedia has a nice example that relates to this question, and the answers to this CV question are also very useful.

The rank of the observation matrix can tell you if it is possible to find a unique solution for your model, which almost never happens. If that were the case and you had a full-rank matrix you could just invert the matrix (for linear problems, and assuming it is square) and use that as you model.

Most ML tasks deal with this kind of over-determined systems, and we use various optimization techniques (such as SGD or coordinate descent) that are designed to provide approximate solutions to over-determined systems.

Community
  • 1
Bar
  • 2,492
  • 3
  • 19
  • 31
  • Why the inverted matrix could be used as the model? In most supervised ML applications, one would like to separate two or more classes of observations from each other. I don't see how the inverse matrix would be useful there. Though, the question does not define if it is about supervised/unsupervised learning nor the number of classes. – mmh May 15 '15 at 12:19
  • @mmh That part of my answer relates to the Wikipedia example, where you have five unknowns m (model) and five equations G. You want to explain the data (d), through the equations (G) and a model (m). You can do that by d = Gm <=> m = G^-1d, which gives you the model parameters, if G is invertible. – Bar May 15 '15 at 12:51