A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models
  with Adversarial Learning
  Vision-language models can encode societal biases and stereotypes, but there
are challenges to measuring and mitigating these harms. Prior proposed bias
measurements lack robustness and feature degradation occurs when mitigating
bias without access to pretraining data. We address both of these challenges in
this paper: First, we evaluate different bias measures and propose the use of
retrieval metrics to image-text representations via a bias measuring framework.
Second, we investigate debiasing methods and show that optimizing for
adversarial loss via learnable token embeddings minimizes various bias measures
without substantially degrading feature representations.
