1

Paraphrased from Understanding Machine Learning by Shalev-Shwartz:

Let $\mathbf{x}_1, \dots, \mathbf{x}_m \in \mathbb{R}^d$, $\mathbf{W}$ an $n \times d$ matrix with real entries, and $\mathbf{U}$ a $d \times n$ matrix with real entries, with $n < d$. The principal component analysis (PCA) problem entails finding $\mathbf{U}$ and $\mathbf{W}$ which minimize the loss function \begin{equation} \sum_{i=1}^{m}\|\mathbf{x}_i - \mathbf{U}\mathbf{W}\mathbf{x}_i\|^{2}_{2}\text{.} \end{equation}

What is the corresponding objective funtion for partial least squares (PLS), if there is one? From what I'm getting from Introduction to Statistical Learning (ISL) by James et al. (anyone who's read this book would know it doesn't go into much mathematical detail), my guess would be if $y_1, \dots, y_m\in \mathbb{R}$, the objective function is finding the $n \times d$ matrix $\mathbf{V}$ and the $n \times 1$ matrix $\boldsymbol\beta$ which minimize
$$\sum_{i=1}^{m}\left[y_i-(\mathbf{V}\mathbf{x}_i)^{T}\boldsymbol\beta\right]^{2}$$ since ISL states:

Like PCR, PLS is a dimension reduction method, which first identifies a new set of features $Z_1, \dots, Z_M$ $(M$ corresponding to $n$ in this case) that are linear combinations of the original features, and then fits a linear model via least squares using these $M$ new features.

Am I interpreting this correctly?

Clarinetist
  • 3,761
  • 3
  • 25
  • 70
  • In your formulation, what stops you from reducing it to a typical regression problem by defining $\gamma \equiv V^T\beta$? – eric_kernfeld Nov 07 '17 at 15:28
  • @eric_kernfeld Ah, that's an excellent point. And I'm completely with you, PLS is extremely confusing. It took me going through 4 books to understand the theory behind PCA, and I can't find any mention of PLS other than in the Stanford texts (ISL, Elements of Statistical Learning). – Clarinetist Nov 07 '17 at 15:32
  • I'm going to list some references to PLS questions here on CV. Have you looked at these? [Q1](https://stats.stackexchange.com/questions/23353/pca-lda-cca-and-pls) [Q2](https://stats.stackexchange.com/questions/300904/dimensionality-reduction-interpretation) [Q3](https://stats.stackexchange.com/questions/206997/probabilistic-models-for-partial-least-squares-reduced-rank-regression-and-can) [Q4](https://stats.stackexchange.com/questions/206587/what-is-the-connection-between-partial-least-squares-reduced-rank-regression-a/206883#206883) – eric_kernfeld Nov 07 '17 at 15:55
  • 1
    Also https://stats.stackexchange.com/questions/179733 – amoeba Nov 07 '17 at 16:01

0 Answers0