Realistic Video Summarization through VISIOCITY: A New Benchmark and
  Evaluation Framework
  Automatic video summarization is still an unsolved problem due to several
challenges. We take steps towards making automatic video summarization more
realistic by addressing them. Firstly, the currently available datasets either
have very short videos or have few long videos of only a particular type. We
introduce a new benchmarking dataset VISIOCITY which comprises of longer videos
across six different categories with dense concept annotations capable of
supporting different flavors of video summarization and can be used for other
vision problems. Secondly, for long videos, human reference summaries are
difficult to obtain. We present a novel recipe based on pareto optimality to
automatically generate multiple reference summaries from indirect ground truth
present in VISIOCITY. We show that these summaries are at par with human
summaries. Thirdly, we demonstrate that in the presence of multiple ground
truth summaries (due to the highly subjective nature of the task), learning
from a single combined ground truth summary using a single loss function is not
a good idea. We propose a simple recipe VISIOCITY-SUM to enhance an existing
model using a combination of losses and demonstrate that it beats the current
state of the art techniques when tested on VISIOCITY. We also show that a
single measure to evaluate a summary, as is the current typical practice, falls
short. We propose a framework for better quantitative assessment of summary
quality which is closer to human judgment than a single measure, say F1. We
report the performance of a few representative techniques of video
summarization on VISIOCITY assessed using various measures and bring out the
limitation of the techniques and/or the assessment mechanism in modeling human
judgment and demonstrate the effectiveness of our evaluation framework in doing
so.
