2

We are trying to generate a model to summarize Persian news. About 14000 news were summarized with help of humans(supervised) and then we extracted all sentences (about 180000) and labeled them (true if were selected in summarization, false if not). We have also calculated 9 features for these sentences (all features are between 0-1). Finally we used MaxEnt classifier (logistic regression) for binary classification on our data set.

And here is the result :
enter image description here

I really don't know low recall and high precision is normal for our classifier or something is wrong in our work? Can anyone come up with an explanation?

mpiktas
  • 33,140
  • 5
  • 82
  • 138
Oli
  • 155
  • 1
  • 7
  • What data do you use to train your classifier? Are you testing it on a separate dataset? – Diego Apr 04 '16 at 19:30
  • no the same dataset.. – Oli Apr 04 '16 at 19:41
  • Then your results, the model itself, is probably not good at all. Try to split the dataset in two and use one part as train and another as test. Then let us know what the precision is. – Diego Apr 04 '16 at 19:47
  • actually we are using k-fold validation with k=10 – Oli Apr 04 '16 at 19:50
  • Hi @mpiktas , I am working on a smiliar task and I was wondering as you mentioned, you balanced your classes. Do you mean that you balanced your training classes or your test classes or both? – noor h Sep 05 '19 at 08:22

1 Answers1

2

Users often prefer higher precision to higher recall. For example, you (and Google) really want the first 15 or 30 results for a search engine query to be accurate (=high precision), but neither of you are particularly concerned if you miss one or two of the millions on pages on the web (=low recall). Obviously, this depends a bit on the application, but I would imagine summarization is similar.

The bad news is that rapid miner has apparently computed precision and recall for each class separately, and it looks like your recall is quite bad (~13%) on the in-summary class, which is presumably more important than the not-in-summary sentences.

I suspect that this is because you have fairly unbalanced classes--it looks like there are ~3x more not-in-summary examples in your data set. You could potentially tweak the logistic regression's decision threshold (probably 0.5 by default) to give extra weight to the positive examples.

Matt Krause
  • 19,089
  • 3
  • 60
  • 101
  • two question:1-is there any good operator in rapidminer to balance our data ? or any good algorithm? 2- there is only C parameter in rapidminer logistic regression operator the default value is 1 how can i change threshold? threshold = C ? – Oli Apr 04 '16 at 19:14
  • I've never actually used rapid miner, but this may do the trick: http://docs.rapidminer.com/studio/operators/modeling/predictive/ensembles/metacost.html There should be some way to change the logistic regression's threshold internally too.... – Matt Krause Apr 04 '16 at 19:25
  • your answer helped me a lot. thanks!! after balancing my dataset recall increased. please see my new question http://stats.stackexchange.com/questions/205684/how-use-maxent-logistic-regression-weights thanksss – Oli Apr 05 '16 at 20:09