1

I am able to implement supervised learning algorithms when the output label ($Y$) is one dimensional. Just in case, the output label is multidimensional ($Y$= [$y_1$ $y_2$ ... $y_s$]), I believe that for such cases, we have to have $s$ different models with $i^{th}$ model predicting $y_i$ from the input data.

Am I right with this interpretation ? Any better alternative ?

Let's assume I am implementing linear regression algorithm on a labelled data with the output label being $s$ dimensional.

Upendra01
  • 1,566
  • 4
  • 18
  • 28

1 Answers1

2

For linear regression, a muti-label output is the same as multiple linear regression models - each has its own coefficients. But for other models, such as a neural network, one model with several outputs is not the same as several independent models. In such a case, if you believe the labels depend on some common underlying structure then it is better to use one model with multiple outputs.

elliotp
  • 670
  • 3
  • 13
  • could you please explain the last line of yours "In such a case ... with multiple outputs" Why not this sentence hold good for linear regression as well. I mean, in linear regression also, y1 and y2 are dependent on 'some common underlying structure' that we call as features. I am getting confused with 'some common underlying structure'. Could you please elaborate on this – Upendra01 Jan 28 '18 at 09:09
  • @UpendraPratapSingh in neural networks, the output layer depends on the last hidden layer, and that layer depends on the layer before it, etc. It is possible for the whole network to learn to represent the data and only the output layer distinguishes between the different outputs. This is not the case in linear regression. – elliotp Jan 28 '18 at 09:24
  • So 'some common underlying structure' refers to the abstract representation of the data at the last hidden layer Right? – Upendra01 Jan 28 '18 at 09:29
  • @UpendraPratapSingh it can be several of the last layers. For example, a convolutional neural network might learn to represent natural images well in a low dimensional space, and then the final two layers learn to classify animals. Cat or not cat would be one output, dog or not dog would be a second output, etc. Most of the work being done is common to all the animals. – elliotp Jan 28 '18 at 09:35
  • Right... but with this explanation of yours, even the features in linear regression are representing the data and the output labels (y1 as well as y2) are dependent on these features (through coefficients). So, now, what is it that allows us to choose separate models for different y's and what makes it absolutely necessary for us to go with a single model with multiple outputs ? I believe that this distinction is an important issue for the entire community of machine learning practitioners. Kindly educate me on this. By the way. thanks for showing a genuine interest in my query. – Upendra01 Jan 28 '18 at 09:43
  • @UpendraPratapSingh when it is one model, you can train it on all the data and learn a common representation for images as well as the supervised task such as classification of different animals. You might not have enough examples of each animal to train separate models. This also benefits the computation cost, both during training and at test time - one model that can perform the same task as many independent models will require fewer computations. – elliotp Jan 28 '18 at 10:05
  • "For linear regression, a muti-label output is the same as multiple linear regression models" This is true from the standpoint of prediction, but no longer true if we care about statistical inference (confidence intervals on the parameters, hypothesis tests, etc.) E.g. see here: https://stats.stackexchange.com/questions/318810/multivariate-linear-regression-vs-several-univariate-regression-models – user20160 Jan 28 '18 at 11:20