5

Say you're given a dataset where the response $y$ is continuous. The only prediction you're after is whether $y \geq C$, i.e., where the response is greater than some value.

In this case, would it be better to

(1) Use a regression algorithm that produces a continuous output, and then manually check if the response is $\geq C$?

(2) Use a classification algorithm. To do this, we must first preprocess the training data so the samples with $y_i \geq C$ are set to $1$ for binary classification and 0 otherwise.

It seems that both are feasible, but which is better, or what are the advantages/disadvantages of both?

David
  • 563
  • 2
  • 6
  • 1
    Regression will tell you not only if it is bigger than C but also how much bigger it is, this is an important piece of information. – user2974951 Jul 29 '20 at 05:42
  • 1
    @user2974951. I'm assuming you don't care about how much bigger it is and just that it's bigger or not. In other words you're given continuous data, but the only thing you care about is a binary answer. – David Jul 29 '20 at 05:44

2 Answers2

4

Interesting question. First of all, classic linear regression was developed for applications where the scatter is normally distributed. If you plot the residual distribution it should have the classic bell shape. When your data adhere to the these model prerequisites, your can just as well use linear regression. The confidence bounds for linear regression predictions are also known, it is advisable that you use these.

When your predictor variables come from discrete, or multimodal continuous distributions then you can benefit from a nonparametric classifier. For example the histogram classifier or the K-nearest neighbor classifier can be used. I presuppose that training a neural network for the prediction of $y$ would be somehow 'over-the-top'.

So my advice is to let your choice be given by the distributions of your data.

Match Maker EE
  • 1,701
  • 4
  • 15
  • Why does nonparametric classifiers work better if the predictor variables come from discrete/multimodal distributions? – David Jul 29 '20 at 17:18
  • For this types of distributions, the the prerequisites of linear regression are clearly not fulfilled. With a histogram classifier, a nonparametric criterion is used to determine the most likely outcome of $y$. – Match Maker EE Jul 29 '20 at 20:30
  • Ah. I've been under the impression that linear regression is compatible with discrete predictors. That seems to not be the case? – David Jul 29 '20 at 23:42
  • You your most important predictor variable is a multimodal distribution with say 3 clear peaks, then linear regression is not an appropriate model to use. – Match Maker EE Jul 30 '20 at 07:24
2

As you write, either approach is feasible. I don't think you can give a general recommendation. In some situations, there may be more existing knowledge pertaining to one than to the other approach - for instance, if you are forecasting a time series, there is much more work on forecasting a continuous target variable (approach 1) than a binary one (approach 2).

So I would recommend that you try both approaches and see which one works better for your problem at hand. Just be sure to use an appropriate measure of "better" - not accuracy, that is.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357