Induction and Reference of Entities in a Visual Story
  We are enveloped by stories of visual interpretations in our everyday lives.
The way we narrate a story often comprises of two stages, which are, forming a
central mind map of entities and then weaving a story around them. A
contributing factor to coherence is not just basing the story on these entities
but also, referring to them using appropriate terms to avoid repetition. In
this paper, we address these two stages of introducing the right entities at
seemingly reasonable junctures and also referring them coherently in the
context of visual storytelling. The building blocks of the central mind map,
also known as entity skeleton are entity chains including nominal and
coreference expressions. This entity skeleton is also represented in different
levels of abstractions to compose a generalized frame to weave the story. We
build upon an encoder-decoder framework to penalize the model when the decoded
story does not adhere to this entity skeleton. We establish a strong baseline
for skeleton informed generation and then extend this to have the capability of
multitasking by predicting the skeleton in addition to generating the story.
Finally, we build upon this model and propose a glocal hierarchical attention
model that attends to the skeleton both at the sentence (local) and the story
(global) levels. We observe that our proposed models outperform the baseline in
terms of automatic evaluation metric, METEOR. We perform various analysis
targeted to evaluate the performance of our task of enforcing the entity
skeleton such as the number and diversity of the entities generated. We also
conduct human evaluation from which it is concluded that the visual stories
generated by our model are preferred 82% of the times. In addition, we show
that our glocal hierarchical attention model improves coherence by introducing
more pronouns as required by the presence of nouns.
