ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.
Questions tagged [rouge]
5 questions
17
votes
2 answers
Interpreting ROUGE scores
I recently read the paper on Salesforce's advances in abstractive text summarisation. This states that the ROUGE-1 score achieved of 41.16 is significantly better than the previous state of the art.
I also read this paper on (mainly extractive) text…
Alan Buxton
- 275
- 1
- 2
- 8
4
votes
1 answer
Gaming the ROUGE metric for text summarization
ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated).
$$ROUGE_{n}= \frac {\sum_{s\in \textrm{Ref Summaries} } \sum_{gram_{n}\in…
wabbit
- 370
- 2
- 13
2
votes
1 answer
Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?
I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures precision, all ROUGE implementations I have come…
jdepoix
- 71
- 2
0
votes
0 answers
ROUGE-N for multiple references
In the original paper (https://aclanthology.org/W04-1013.pdf) that presented ROUGE the description of ROUGE-N is rather unclear on the case when multiple references are used.
Firstly it presents the ROUGE-N as
$$
\text{ROUGE-N} = \frac{\sum_{S \in…
Martin
- 11
- 2
0
votes
0 answers
Why isn't ROUGE-N normalized by the number of N-grams in the reference summary?
Note: I'll focus on $ROUGE-1$, but the same holds for $ROUGE-N$.
For a machine-produced summary $M$ and a bunch of reference summaries $RefSummaries$, I believe $ROUGE-1$ can be calculated in the following manner:
$$ROUGE\textrm{-}1 =…
asymmetryFan
- 101
- 2