Dependent Multi-Task Learning with Causal Intervention for Image
  Captioning
  Recent work for image captioning mainly followed an extract-then-generate
paradigm, pre-extracting a sequence of object-based features and then
formulating image captioning as a single sequence-to-sequence task. Although
promising, we observed two problems in generated captions: 1) content
inconsistency where models would generate contradicting facts; 2) not
informative enough where models would miss parts of important information. From
a causal perspective, the reason is that models have captured spurious
statistical correlations between visual features and certain expressions (e.g.,
visual features of "long hair" and "woman"). In this paper, we propose a
dependent multi-task learning framework with the causal intervention (DMTCI).
Firstly, we involve an intermediate task, bag-of-categories generation, before
the final task, image captioning. The intermediate task would help the model
better understand the visual features and thus alleviate the content
inconsistency problem. Secondly, we apply Pearl's do-calculus on the model,
cutting off the link between the visual features and possible confounders and
thus letting models focus on the causal visual features. Specifically, the
high-frequency concept set is considered as the proxy confounders where the
real confounders are inferred in the continuous space. Finally, we use a
multi-agent reinforcement learning (MARL) strategy to enable end-to-end
training and reduce the inter-task error accumulations. The extensive
experiments show that our model outperforms the baseline models and achieves
competitive performance with state-of-the-art models.
