1

I trained a different number of classifiers on a problem (logistic regression, KNN, neural networks, svm...). The training happened with double cross validation and all the gold standards, so I'm pretty confident they are good classifiers, in the sense that they do not overfit. If I examine the confusion matrix each one of them generates on a test set, I see they all make almost the same mistakes (numbers in the confusion matrix are roughly the same for each one of them).

Can this common pattern in how they classify be exploited in some way? For example, I was thinking of stacking one of such models with other models that have decorrelated predictions, something along these lines. Does this make any sense? Do you have better ideas to propose?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
mp85
  • 549
  • 2
  • 10

1 Answers1

0

See https://en.wikipedia.org/wiki/Boosting_%28machine_learning%29

You are close to rediscovering boosting, an ensemble learning idea consisting in learning multiple weak/simple algorithms, but reweighting the data, so the examples badly classified for the first algorithms, are given higher weigh, so later algorithms maybe can find patterns in the data that help with difficult examples. Search this site for boosting.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • Yeah, I'm aware of gb, but actually it performs like the others. I'm thinking that I sort of hit the "upper bound" of performance, given that the outcome that I'm predicting is also determined by features that are not included on purpose (it's a medical problem, so I was told not to use certain attributes). – mp85 Nov 03 '16 at 22:21