Assessing Multilingual Fairness in Pre-trained Multimodal
  Representations
  Recently pre-trained multimodal models, such as CLIP, have shown exceptional
capabilities towards connecting images and natural language. The textual
representations in English can be desirably transferred to multilingualism and
support downstream multimodal tasks for different languages. Nevertheless, the
principle of multilingual fairness is rarely scrutinized: do multilingual
multimodal models treat languages equally? Are their performances biased
towards particular languages? To answer these questions, we view language as
the fairness recipient and introduce two new fairness notions, multilingual
individual fairness and multilingual group fairness, for pre-trained multimodal
models. Multilingual individual fairness requires that text snippets expressing
similar semantics in different languages connect similarly to images, while
multilingual group fairness requires equalized predictive performance across
languages. We characterize the extent to which pre-trained multilingual
vision-and-language representations are individually fair across languages.
However, extensive experiments demonstrate that multilingual representations do
not satisfy group fairness: (1) there is a severe multilingual accuracy
disparity issue; (2) the errors exhibit biases across languages conditioning
the group of people in the images, including race, gender and age.
