Classify and regress at the same time

Question

Is it possible to have a model that aims to classify and regress at the same time?

For example if I have five independent variables and I want to use these 5 variables to predict the gender of the observation. Subsequently I let the predicted gender be the 6th independent variable and predict for income of the individual.

How I train the model is by splitting the dataset into two parts. The first part is used to train the model to predict gender, income is ignored here. The second part is used to train the model to predict the income of the individual based on all the other variables including gender.

Is this methodology sound? If not, why so?

Answer 1

Answering the specific question.

Yes, it is possible (but might be undesirable)

Directed graph models (such as ANNs) can cope with that.

You have 5 input variables, and you want to predict gender first, and include this prediction to predict income.

Basically, you need to connect all your inputs to an output, which is gender, then connect all your inputs again to a second output, income (a skip connection, in ANN terminology), plus your first output to the second. Also, don't forget bias terms.

Arrows are weights (coefficients) and circles are nodes.

Then, you want to minimize binomial loss on the prediction of gender, and (perhaps) squared loss on the prediction of income, optimizing the weights in the model. You can write this as your loss function.

$$\mathbb L_\text{total} (\text{input, gender, income})=\mathbb L _\text{binomial}(\text{gender})+\mathbb L _\text{squared}(\text{income})$$

You might want to normalize the loss terms, because, due to variance, one might dominate over the other.

Notice that the prediction of income depends on all weights in the model, so optimizing for the prediction of gender separately first might not be optimal to the predicition of income.

---

Now,

While this is specifically what you asked for, you have to ask if it's really useful. The separate prediction of gender and subsequently income including the gender predictions might be simpler, and above all, easier to implement.

---

Also, on imputation,

While studying these mediating relationships might be of interest, I have to warn you about the possibility of imputation.

If your objective is to account for missing gender information in some observations, then consider that the model won't use the actual gender information of the observations containing it. If the independent variables do not predict gender reasonably, you'll be basically forfeiting this information in your model, probably tending to a no informative output in the gender node.

Answer 2

Why do you want to do this? For your proposal to be possible and meaningful, it must be that you have some missing observations on the binary gender predictor (but not all). If that is the case, the usual approach is to use some (multiple) imputation method. See Multiple imputation for missing values or search this site!

(see also the comments for other ideas). There are some papers about imputation for binary (or categorical) predictors, for instance https://www.statalist.org/forums/forum/general-stata-discussion/general/29284-multiple-imputation-for-missing-categorical-variables, https://statisticalhorizons.com/ml-better-than-mi, https://www.sciencedirect.com/science/article/pii/S0167947310001490

(Sorry if I have misinterpreted the question, then it needs clarification)

Answer 3

As the other answers pointed out, this may not be something you want to do. But in cases where your 2 regression targets are closely related (say in the case of latitude and longitude) it may make sense.

One of the things I really love about neural networks is that they are very flexible, and allow you to define one model that does multiple things. For example, you can have one model that predicts both a classification target and a regression target.

Even if that's not what you should do on this dataset, I still think it's pretty cool, so I'm gonna share some code for a keras neural network in python that does classification and regression at the same time!

First, lets load some data. In this case, we're gonna predict sex (classification) and age (regression) from some census data:

import pandas as pd

adult = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data')
adult.columns = ['age', 'workclass', 'fnlwgt', 'edu', 'edu_num', 'marital_status',
                 'occupation', 'relationship', 'race', 'sex', 'capital_gain', 'capital_loss',
                 'hours_per_week', 'native_country', 'income']
adult["sex"] = adult["sex"].astype('category').cat.codes
adult["workclass"] = adult["workclass"].astype('category').cat.codes
adult["marital_status"] = adult["marital_status"].astype('category').cat.codes
adult["race"] = adult["race"].astype('category').cat.codes
adult["occupation"] = adult["occupation"].astype('category').cat.codes
adult["native_country"] = adult["native_country"].astype('category').cat.codes

target_bin = adult[['sex']].values.astype('float32')
target_num = adult[['age']].values.astype('float32')
X = adult[['workclass', 'edu_num', 'marital_status', 'occupation', 'race', 'capital_gain',
           'capital_loss', 'hours_per_week', 'native_country']].values.astype('float32')

X is our predictors, target_bin is the classification target, and target_num is the regression target.

Now lets define a keras model with one input (X) and 2 outputs (target_bin and target_num): from keras.layers import Input, Dense, BatchNormalization from keras.models import Model from keras.optimizers import Adam from keras.utils.vis_utils import plot_model

input = Input(shape=(X.shape1,), name='Input') hidden = Dense(64, name='Shared-Hidden-Layer', activation='relu')(input) hidden = BatchNormalization()(hidden) out_bin = Dense(1, name='Output-Bin', activation='sigmoid')(hidden) out_num = Dense(1, name='Output-Num', activation='linear')(hidden)

model = Model(input, [out_bin, out_num]) model.compile(optimizer=Adam(0.10), loss=['binary_crossentropy', 'mean_squared_error']) model.summary() plot_model(model, to_file='model.png')

Now we can fit this model to our data:

model.fit(X, [target_bin, target_num], validation_split=.20, epochs=100, batch_size=2048)
model.predict(X)

We use a 20% validation set, where the model gets a logloss of 0.5649 and a mean squared error of 117.0967, which corresponds to an RMSE of 10.8. A logloss of <.69 means the model learned something useful for classification, and the RMSE of 10.8 is pretty good given that the mean of the numeric target is 38.6.

If you look at this model's predictions, you see it predicts 2 arrays. The first is the probability that the person is female, and the second is the person's predicted age.

You'll notice that I setup this network to use a shared hidden layer from the inputs to predict the outputs. You could also setup the network to e.g. use the sex prediction as a predictor for age, or other creative ways to relate the 2 variables.

So you can make one model to do both classification and regression!