If you have a matrix with n rows and m columns, you can use SVD or other methods to calculate a low-rank approximation of the given matrix.
However, the low rank approximation will still have n rows and m columns. How can low-rank-approximations be useful for machine learning and natural language processing, given that you are left with the same number of features?
A low rank approximation $\hat{X}$ of $X$ can be decomposed into a matrix square root as $G=U_{r}\lambda_{r}^\frac{1}{2}$ where the eigen decomposition of $X$ is $U\lambda U^T$, thereby reducing the number of features, which can be represented by $G$ based on the rank-r approximation as $\hat{X}=GG^T$. Note that the subscript $r$ represents the number of eigen-vectors and eigen-values used in the approximation. Hence, it does reduce the number of features to represent the data. In some examples low-rank approximations are considered as basis or latent variable (dictionary) based expansions of the original data, under special constraints like orthogonality, non-negativity (non-negative matrix factorization) etc.
The point of low-rank approximation is not necessarily just for performing dimension reduction.
The idea is that based on domain knowledge, the data/entries of the matrix will somehow make the matrix low rank. But that is in the ideal case where the entries are not affected by noise, corruption, missing values etc. The observed matrix typically will have much higher rank.
Low-rank approximation is thus a way to recover the "original" (the "ideal" matrix before it was messed up by noise etc.) low-rank matrix i.e., find the matrix that is most consistent (in terms of observed entries) with the current matrix and is low-rank so that it can be used as an approximation to the ideal matrix. Having recovered this matrix, we can use it as a substitute for the noisy version and hopefully get better results.
Two more reasons not mentioned so far:
Reducing colinearity. I believe that most of these techniques remove colinearity, which can be helpful for follow-on processing.
Our imaginations are low-rank, so it can be helpful for exploring low-rank relationships.
Once you have decided the rank of the approximation(say $r<m$) , you will only retain the $r$ basis vectors for future use (say, as predictors in a regression or classification problem) and not the original $m$.
According to "Modern multivariate statistical techniques (Izenman)" reduced rank regression covers several interesting methods as special cases including PCA, factor analysis, canonical variate and correlation analysis, LDA and correspondence analysis
