Reconstruction error in PCA?

Question

I'm using PCA for a while, but recently I read about reconstruction error which I cannot understand...

For example let's consider dataset consisting 5 variables: $X_1, X_2, X_3, X_4, X_5$. Additionally I obtained two variables $Z_1$ and $Z_2$ by using principal components analysis using two components which are linear combinations of $X_i$, $i = 1,2,3,4,5$. What in this case I should perceive as a reconstruction error?

The most logical thing for me would be:

Calculate RMSE of $Z_1$ with $X_1, X_2, X_3, X_4$ and $X_5$. Output is then 5 numbers.
Calculate RMSE of $Z_2$ with $X_1, X_2, X_3, X_4$ and $X_5$. Output is also consisting 5 numbers.

What I would say is that reconstruction error is mean of those 10 numbers. Am I correct?

Your reading source ought to have a mathematical definition, or at least a clear description, of what it means by "reconstruction error:" what is it? — whuber, Dec 03 '21 at 16:25
Relevant questions https://stats.stackexchange.com/questions/194278/meaning-of-reconstruction-error-in-pca-and-lda and https://stats.stackexchange.com/questions/438716/reconstruction-error-in-pca — msuzen, Dec 04 '21 at 11:31

DifferentialPleiometry · Accepted Answer · 2021-12-03T17:39:15.220

I have seen this term "reconstruction error" in the context of PCA before.

I will skim over most of the details of PCA, but I recommend you become adequately familiar with diagonalization and SVD in the context of standard linear algebra (if you are not already).

One of the things that PCA provides is a transformation of the data in the form of a change in basis. This change in basis is invertible.

When performing PCA for the purpose of dimensionality reduction, a subset of the principal components are chosen either for exploratory data analysis or as machine learning features.

Taking only a subset of your principal components, and the corresponding inverse change in basis, you can convert your transformed variables back to the original space of your data.

If you kept all of the principal components in the inverse transformation, you should perfectly reproduce your data. In reality there can be some floating point error, an issue with your computer keeping track of only a finite number of decimal places, but this is not what we mean by "reconstruction error" in this context.

Rather, if you apply the inverse transformation using only a subset of your principal components, you probability will not reproduce the original dataset perfectly.

Computing a loss function, such as RMSE or similar functions, between the original data and the reconstruction of the data yields a number that we call "reconstruction error".

Here is an illustration of the process, if it helps your understanding.

Code for the diagram, if you want to use or modify it.

from graphviz import Digraph

g = Digraph('stack')

g.node('Reconstruction\nError',shape='square')

g.edge('Transformed\nData',
       'Data',
       label='Inverse\nChange\nof\nBasis')
g.edge('Data',
       'Transformed\nData',
       label='Change\nof\nBasis')
g.edge('Transformed\nData',
       'Dimensionality\nReduced\nTransformed\nData',
       label='Select\nComponents')
g.edge('Dimensionality\nReduced\nTransformed\nData',
       'Lossy\nReconstructed\nData',
       label='Inverse\nChange\nof\nBasis')
g.edge('Lossy\nReconstructed\nData',
       'Reconstruction\nError',
       label='RMSE',
       style='dashed')
g.edge('Data',
       'Reconstruction\nError',
       label='RMSE',
       style='dashed')
g.view()

1

(+1) Nice use of graphviz. – msuzen Dec 04 '21 at 11:26

Reconstruction error in PCA?

1 Answers1