It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image
  Captioning by Contrastive Data Collection
  Datasets that capture the connection between vision, language, and affection
are limited, causing a lack of understanding of the emotional aspect of human
intelligence. As a step in this direction, the ArtEmis dataset was recently
introduced as a large-scale dataset of emotional reactions to images along with
language explanations of these chosen emotions. We observed a significant
emotional bias towards instance-rich emotions, making trained neural speakers
less accurate in describing under-represented emotions. We show that collecting
new data, in the same way, is not effective in mitigating this emotional bias.
To remedy this problem, we propose a contrastive data collection approach to
balance ArtEmis with a new complementary dataset such that a pair of similar
images have contrasting emotions (one positive and one negative). We collected
260,533 instances using the proposed method, we combine them with ArtEmis,
creating a second iteration of the dataset. The new combined dataset, dubbed
ArtEmis v2.0, has a balanced distribution of emotions with explanations
revealing more fine details in the associated painting. Our experiments show
that neural speakers trained on the new dataset improve CIDEr and METEOR
evaluation metrics by 20% and 7%, respectively, compared to the biased dataset.
Finally, we also show that the performance per emotion of neural speakers is
improved across all the emotion categories, significantly on under-represented
emotions. The collected dataset and code are available at
https://artemisdataset-v2.org.
