Using machine learning and having a data set with a target that can be both seen as a numerical value or a class, how could you compare the outcomes of the two possible type of models.
For example:
Model A sees the target as a class and makes predictions that have an accuracy of 72%.
Model B sees the target as a numerical value and end up having an R^2 of 0.4646.
How would you explain in this case what the best model too choose is, model A or model B?
I assume that model B outputs numerical values between 0 and 1. These can then be interpreted as probabilities for the instance to belong to the target class, i.e., to have a true value of 1.
Then I would choose model B. Not because of the specific values of accuracy or of $R^2$, but because probabilistic predictions are inherently better than hard classifications.
For my reasoning why I think so, please take a look at my answers to Reduce Classification Probability Threshold and Why is accuracy not the best measure for assessing classification models?.
