0

What is the geometrical shape of the resulting function when you do a multivariate linear regression (aka General linear model)? (roughly speaking: Y = f(X) = A.X where f is the resulting function, A is a matrix and X, Y vectors)

Firstly, I thought that it was always a line in a r dimensional space (r = input + output dimension). But if you work in 3D, with real inputs of size 2 (X=(x1, x2)) and real outputs to predict with size 1 (y), it seems that you obtain a plane (f(X) = f(x1, x2) = a.x1 + b.x2 = y).

With inputs of $n$ dimensions and outputs of $m$ dimensions, I find that you obtain the equation of $m$ hyperplanes in a $r=m+n$ space (or m linear equations). So you have a line only if you have n=1 (I didn't take into account the special case where there are collinear hyperplanes). So in most cases, you find the resultant function is a linear subspace of size n (since the intersection of m hyperplanes has a dimension of r-m = n+m-m = n).

Am I right about that?

I'm curious but, is it possible to train a multivariate regression model (i.e. a training model where independent and dependant variables are vectors, not scalars) with always a line shape? (for me, it seems theoretically impossible)

I use sklearn.linear_model.LinearRegression to do my multivariate linear regression (which seems to be ok).

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
ArnoNoo
  • 1
  • 1
    Multivariate linear regression GLM (or MAN(C)OVA, whenever categorical predictors are present) is almost a synonym to canonical correlation analysis (CCA). Wrt fitting algorithm they are same, difference are in accents: in any regression, we pay different attention and compute different statistics for dependent and independent variables, in correlation, we usually are not interested "asymmetric" testing accompanying all that. CCA is the latent framework of GLM (general linear model). – ttnphns Sep 03 '17 at 18:07
  • 1
    (cont.) There is decent number of threads on this site about it (so search). Since you want geometry, I recommend [my answer explaining CCA](https://stats.stackexchange.com/a/65817/3277) for initial acquaintance. One more [answer](https://stats.stackexchange.com/a/169483/3277) of mine stresses the close relationship of MANOVA and linear discriminant analysis (which is equaivalent to CCA with dummy variables). – ttnphns Sep 03 '17 at 18:07
  • 1
    One this site you should use $\LaTeX$-notation for mathematics, please edit (I added some) – kjetil b halvorsen Sep 10 '17 at 17:30

1 Answers1

0

I'm not sure to understand (deeply) correctly the statistical perspective of my problem, but I think that I have an answer (and maybe I can explain it a little more clearly): 1) Yes, I think I'm right: you find always the equation of linear subspace of size n which tries globally to get as close as possible of all the vectors (training set) in the whole space (size n+m). These vectors being constructed in concatenating your training input and output vectors. 2) Indeed, it is not really possible to have the equation of a line. If you have a set of training vectors (size n+m), naturally, you can look for the line which approches globally the best all yours vectors. But you need to predict output vectors in using input vectors of size n. So, it exist a lot of points which are even not in the domain of your function. Thus, it is a nonsense to want to approximate your resulting model function by a line!

ArnoNoo
  • 1