0

I'm familiar with the closed-form solution of ordinary least squares which minimizes $\sum_{n=1}^N(y_n - \mathbf{\beta x_n})^2$ for scalar $y_n$. However, in my situation I am trying to fit some data to a vector output through a matrix transformation, so I am instead looking to minimize the norm of the difference for each sample:

$$\min_B\sum_{n=1}^N\|\mathbf{y_n}-B\mathbf{x_n}\|^2$$

where $\mathbf{y_n}$ is now a vector and the parameters are now a matrix $B$. I suspect I am just unfamiliar with the name for this as searches of "vector response OLS" and "matrix OLS" keep bringing me the compact matrix formulation of OLS rather than the problem above.

concat
  • 111
  • 3
  • 4
    The key search term is "multivariate." Many people confuse this with "multiple regression," so eliminate "multiple," include "squares" to capture the OLS flavor of your problem, and check out the hits: https://stats.stackexchange.com/search?q=multivariate+regression+-multiple+squares. In light of this, what is your question? – whuber Feb 20 '19 at 17:04
  • 1
    The model you're looking for is called "[Multivariate Multiple Regression](https://stats.stackexchange.com/questions/4517/regression-with-multiple-dependent-variables)" – Lucas Farias Feb 20 '19 at 17:07

1 Answers1

1

Ah, I just realized that I can run OLS on each output variable separately with the same input data and concatenate the resulting parameter vectors into $B$. The cost function is separable into the OLS cost of each output variable:

$$\|\mathbf{y_n}-B\mathbf{x_n}\|^2=\sum_{k=1}^{M}(y_{n,k}-\mathbf{B_k\mathbf{x_n})^2}$$

where $\mathbf{B_k}\in \mathbb{R}^{1\times D}$ is the k-th row of $B$. OLS minimizes each term, and the total will be minimized because the parameters of each term are independent.

concat
  • 111
  • 3
  • This is correct. However, in practice it often is too much to suppose that you really want to minimize the sum of squares: typically you need to weight the different variables differently, perhaps by unknown amounts; and not infrequently, you also have to account for nonzero correlations among them: that's what makes it truly a multivariate problem. – whuber Feb 20 '19 at 19:07