regression with scikit-learn with multiple outputs, svr or gbm possible?

Question

I have been trying regression with scikit-learn with a problem with multiple outputs like this:

X = np.random.random((10,3))
y = np.random.random((10,2))
X2 = np.random.random((7,3))
clf = SVR()
clf.fit(X, y)
y_pred = clf.predict(X2)

The problem is that this doesn't work. It fails with:

ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

Does anyone know how to deal with regression with multiple outputs in scikit-learn?

Edit. I have noticed RandomForestRegressor, KNeighborsRegressor, and LinearRegression all work, but rf is the only one that's close to being good on my dataset! Is there some way to fit SVR or gbm?

score 8 · Accepted Answer · answered May 25 '15 at 11:35

Why not make a wrapper that would fit m regressors (where m is dimensionality of each y) like this?

class VectorRegression(sklearn.base.BaseEstimator):
    def __init__(self, estimator):
        self.estimator = estimator

    def fit(self, X, y):
        n, m = y.shape
        # Fit a separate regressor for each column of y
        self.estimators_ = [sklearn.base.clone(self.estimator).fit(X, y[:, i])
                               for i in range(m)]
        return self

    def predict(self, X):
        # Join regressors' predictions
        res = [est.predict(X)[:, np.newaxis] for est in self.estimators_]
        return np.hstack(res)

Note: I haven't tested this code, but you got the idea.

Thanks, i was thinking about something like that.. The only downfall is the problem has 111 outputs! At 1 minute each that would take almost 2 hours! I suppose the list comprehensions could be parallelized here though.. — anthonybell, May 26 '15 at 03:53

Archie · Answer 2 · 2018-02-20T09:26:10.990

Scikit-Learn also has a general class, MultiOutputRegressor, which can be used to use a single-output regression model and fit one regressor separately to each target.

Your code would then look something like this (using k-NN as example):

from sklearn.neighbors import KNeighborsRegressor
from sklearn.multioutput import MultiOutputRegressor

X = np.random.random((10,3))
y = np.random.random((10,2))
X2 = np.random.random((7,3))

knn = KNeighborsRegressor()
regr = MultiOutputRegressor(knn)

regr.fit(X,y)
regr.predict(X2)

score 1 · Answer 3 · answered Aug 08 '21 at 14:33

To answer the question from the edit. I guess that algorithms that naturally support MultiOutput targets, perform best. This is because these algorithms calculate the multiple output variables simultaneously and hence take possible correlations between outputs into account. This is not the case, if you use MultiOutputRegressor from sklearn which fits a model for each output variable individually.

SVR naturally only supports single-output regression. But there are different adaptions that can be made to make the algorithm fit also to a multi-output regression task. For an extensive overview check the paper in the Reference section of this repository.

You can find an example for an implementation of Multiple-output support vector regression in python here. It is based on the paper Multi-step-ahead time series prediction using multiple-output support vector regression.

You also might want to check out this answer.

score -1 · Answer 4 · answered May 25 '15 at 01:41

-1

I think that scikit-learn only supports multi-output regressors in decision tress: DecisionTreeRegressor.

answered May 25 '15 at 01:41

Javier de la Rosa

99
1

Yes, I have noticed that only RandomForestRegressor and LinearRegression seem to work out of the box for multiple output regression. There isn't some class to fit single output regressors into multiple output regressors? Where you can put in any regressor and it will internally call it for each output? – anthonybell May 25 '15 at 02:06

regression with scikit-learn with multiple outputs, svr or gbm possible?

4 Answers4

Linked