Aligning where to see and what to tell: image caption with region-based
  attention and scene factorization
  Recent progress on automatic generation of image captions has shown that it
is possible to describe the most salient information conveyed by images with
accurate and meaningful sentences. In this paper, we propose an image caption
system that exploits the parallel structures between images and sentences. In
our model, the process of generating the next word, given the previously
generated ones, is aligned with the visual perception experience where the
attention shifting among the visual regions imposes a thread of visual
ordering. This alignment characterizes the flow of "abstract meaning", encoding
what is semantically shared by both the visual scene and the text description.
Our system also makes another novel modeling contribution by introducing
scene-specific contexts that capture higher-level semantic information encoded
in an image. The contexts adapt language models for word generation to specific
scene types. We benchmark our system and contrast to published results on
several popular datasets. We show that using either region-based attention or
scene-specific contexts improves systems without those components. Furthermore,
combining these two modeling ingredients attains the state-of-the-art
performance.
