1

I have a text classification task with multiple classes. Right now model uses logistic regression & uses only word features encoded using TF-IDF. But for some of the classes, the non-text features are very useful, such as, length of the text in words, etc.

My question is how it's better to include such features into model, together with text features - how to normalize values, because text length has completely different range of values, comparing to the encoded text features, etc.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Alex Ott
  • 111
  • 5
  • Similar Q: https://stats.stackexchange.com/questions/240869/how-to-use-more-features-in-text-based-machine-learning-models-beyond-the-text-i – kjetil b halvorsen Aug 03 '19 at 18:44

1 Answers1

1

I usually standardize all my features to mean 0 and standard deviation 1. However, I usually work ONLY with non-word features (I work in text readability which means most features I use are structural rather than lexical and the few lexical features are aggregates). Perhaps such standardization might be unstable with a TF-IDF language model?

Johan Falkenjack
  • 455
  • 2
  • 11
  • Yes, this is my concern as well - I understand how to implement either word features, or non-word features, but not both in the same model. – Alex Ott Dec 30 '16 at 19:03
  • How about training two models? One with a TF-IDF language model and one with the non-word features. With logits your outputs are probabilities so you just need to compute a weighted average to combine them, right? – Johan Falkenjack Dec 30 '16 at 21:06
  • Yes, thought about s well, but the averaging may not be the best way to combine... Anyway, thanks for comments - will play with different approaches on smaller models – Alex Ott Dec 31 '16 at 17:09
  • Yeah, you could combine with another layer using the output from both models (with text and without). Ideally you'd optimize all layers at once, as neural nets do (the pooling layer would combine here, and you could have consistent outputs with sigmoid, etc.) – shf8888 Jun 14 '17 at 16:27