Rethinking the Form of Latent States in Image Captioning
  RNNs and their variants have been widely adopted for image captioning. In
RNNs, the production of a caption is driven by a sequence of latent states.
Existing captioning models usually represent latent states as vectors, taking
this practice for granted. We rethink this choice and study an alternative
formulation, namely using two-dimensional maps to encode latent states. This is
motivated by the curiosity about a question: how the spatial structures in the
latent states affect the resultant captions? Our study on MSCOCO and Flickr30k
leads to two significant observations. First, the formulation with 2D states is
generally more effective in captioning, consistently achieving higher
performance with comparable parameter sizes. Second, 2D states preserve spatial
locality. Taking advantage of this, we visually reveal the internal dynamics in
the process of caption generation, as well as the connections between input
visual domain and output linguistic domain.
