Principal Component analysis (vector space or inner product space?)

Question

(WARNING: This question might seem dumb)

I see that the optimization problem in PCA involves the notion of inner product. For example, to solve for the loadings in second principal component, the constraint is that loadings should be orthogonal to loadings from first principal component. Since the notion of inner product (for orthogonality) is involved, I would expect the observations to lie in an inner product space or Hilbert Space. But all text books state that the observations lie in Vector space. My knowledge from functional analysis is that vector spaces are not endowed with inner products and only inner product spaces (and Hilbert spaces) are endowed with notion of inner products.

So when text books state that the observations lie in vector space, do they mean that they lie in inner product space (or Hilbert space)? Or am I missing something?

Thank you.

Answer 1

I agree with F Tusell, Statisticians try to avoid these questions since they are not relevant to the field. But, in my opinion, Statistics in general and PCA in particular are conducted in Hilbert Spaces. Here is why:

In statistics you're working in a vector space with variance/covariance as defined dot product. Uncorrelated variables have Pearson corelation coefficient 0 because the covariance is zero or in other words, their dot product is 0 meaning orthogonality.

Does Variance/covariance satisfy the properties imposed on the dot product in a Hilbert space?

Yes, it does:

• Commutativity: $\operatorname{Cov}(X,Y) = \operatorname{Cov}(Y,X)$
• Linearity: $\operatorname{Cov}(aX+bY,cW+dV) = ac\operatorname{Cov}(X,W) + ad\operatorname{Cov}(X,V) + bc\operatorname{Cov}(Y,W) + bd\operatorname{Cov}(Y,V)$
• Norm is always real valued, positive and fininte: $\operatorname{Cov}(X,X) = \operatorname{Var}(X,X) = \sigma^2$ (standard dev squared) > 0 real and finite
• The Cauchy–Schwarz inequality holds: $\operatorname{Cov}(X,Y) <= \operatorname{Var}(X,X) \operatorname{Var}(Y,Y)$

So a vector space formed by a set of finite dimensional random variables equiped with Variance/Covariance as dot product is indeed a Hilbert Space.

Answer 2

I think most text books take it for granted the inner-product and consider observations as vectors in a vector space with that inner-product. The notion of Hilbert space, on the other hand, involve completeness (containing the limits of all Cauchy sequences of vectors in the space) which in turn guarantees existence and unicity of projections. I think this is also taken for granted in most treatments of PCA.

Simply, multivariate statistics books tend to avoid these details that are not essential for the comprehension of the method nor its succesful application. But I think you are right in what you say if you want to go to that level of detail.