Integrating Image Captioning with Rule-based Entity Masking
  Given an image, generating its natural language description (i.e., caption)
is a well studied problem. Approaches proposed to address this problem usually
rely on image features that are difficult to interpret. Particularly, these
image features are subdivided into global and local features, where global
features are extracted from the global representation of the image, while local
features are extracted from the objects detected locally in an image. Although,
local features extract rich visual information from the image, existing models
generate captions in a blackbox manner and humans have difficulty interpreting
which local objects the caption is aimed to represent. Hence in this paper, we
propose a novel framework for the image captioning with an explicit object
(e.g., knowledge graph entity) selection process while still maintaining its
end-to-end training ability. The model first explicitly selects which local
entities to include in the caption according to a human-interpretable mask,
then generate proper captions by attending to selected entities. Experiments
conducted on the MSCOCO dataset demonstrate that our method achieves good
performance in terms of the caption quality and diversity with a more
interpretable generating process than previous counterparts.
