Using Natural Sentences for Understanding Biases in Language Models
  Evaluation of biases in language models is often limited to synthetically
generated datasets. This dependence traces back to the need for a prompt-style
dataset to trigger specific behaviors of language models. In this paper, we
address this gap by creating a prompt dataset with respect to occupations
collected from real-world natural sentences present in Wikipedia. We aim to
understand the differences between using template-based prompts and natural
sentence prompts when studying gender-occupation biases in language models. We
find bias evaluations are very sensitive to the design choices of template
prompts, and we propose using natural sentence prompts for systematic
evaluations to step away from design choices that could introduce bias in the
observations.
