Efficiency Metrics for Data-Driven Models: A Text Summarization Case
  Study
  Using data-driven models for solving text summarization or similar tasks has
become very common in the last years. Yet most of the studies report basic
accuracy scores only, and nothing is known about the ability of the proposed
models to improve when trained on more data. In this paper, we define and
propose three data efficiency metrics: data score efficiency, data time
deficiency and overall data efficiency. We also propose a simple scheme that
uses those metrics and apply it for a more comprehensive evaluation of popular
methods on text summarization and title generation tasks. For the latter task,
we process and release a huge collection of 35 million abstract-title pairs
from scientific articles. Our results reveal that among the tested models, the
Transformer is the most efficient on both tasks.
