2

I tried many different text classification models from scikit learn. I trained the model using some posts from personal finance stack exchange. Posts are classified into the following four classes: "mortgage", "investing", "credit-card", and "taxes". The model generally works great when tested using the test dataset (I carved out 20% of the data for testing purpose). But if I try something that is totally unrelated to personal finance ("where is the restroom?", for example), the model still classified it into "investing". The problem is the model always picks one out of the four classes doesn't matter how irrelevant the text is. The probabilities the model assigns to the four classes always add up to 1.0. There is always one the wins out. Is there any way to tune the classifier/model so that in the case when the input is irrelevant to any of the classes, all four probabilities are low (they don't add up to 1.0)?

Thanks, Ryan

RyanDing
  • 121
  • 2

1 Answers1

3

Your model only learns about the world from the training data, so in a sense the four classes in your training data represent your model's view of the entire universe. It believes that those four classes are all that exists in the world.

To solve this, you need to teach your model that there is more to the universe than just "mortgage", "investing", "credit-card", and "taxes" (which, to me at least, sounds like a miserable universe to live in). Include some examples of other text in your training data, label it "other", and re-train your model.

Matthew Drury
  • 33,314
  • 2
  • 101
  • 132
  • Just found my question is similar to https://stats.stackexchange.com/questions/83579/multi-class-text-classification-with-a-negative-class. The problem is that the "other" class is unbounded and it hard to find data for the training purpose. – RyanDing Jan 10 '18 at 19:32