Understanding the decomposed matrices in Singular Value Decomposition

Question

I am trying to get some intuition regarding the U and V matrices in SVD ($M=UEV^T$). I think these are orthonormal basis vectors, but I am struggling to get an intuition if they represent anything more than matrix transformations.

For example, in NMF, I understand that the decomposed matrices can be combined as a linear combination of basis vectors in one matrix with weights in the other matrix. In PCA, I understand that the eigenvectors have real significance in representing orthogonal axes of maximal variation, defined by the eigenvalues. But in SVD, I don't see any immediate connection.

Could someone enlighten me?

Answer 1

In a singular value decomposition, $M$ is not (necessarily) symmetric or even square; it's a transformation from one space ($\mathbb{R}^M$) to another ($\mathbb{R}^n$).

Let's supppose $m>n$. $M$ can be decomposed into a transformation into a convenient basis for $\mathbb{R}^n$ by $U$, then a projection and scaling by $E$ into a basis for $\mathbb{R}^n$ , then a rotation into the target basis in $\mathbb{R}^n$ by $V^T$.

(Alternatively, for the compact SVD, $U$ is $m\times n$ and includes the projection, and $E$ is just the scaling)

Also, just as in PCA, it's no loss of generality (and is standard) to organise $E$ from largest to smallest singular value, and $U$ and $V$ represent orthogonal axes of maximal variation in the two spaces. There are various efficient algorithms for giving you just the largest few singular values and corresponding vectors (either Lanczos-type or stochastic algorithms)