Capturing Failures of Large Language Models via Human Cognitive Biases
  Large language models generate complex, open-ended outputs: instead of
outputting a single class, they can write summaries, generate dialogue, and
produce working code. In order to study the reliability of these open-ended
systems, we must understand not just when they fail, but also how they fail. To
approach this, we draw inspiration from human cognitive biases -- systematic
patterns of deviation from rational judgement. Specifically, we use cognitive
biases to (i) identify inputs that models are likely to err on, and (ii)
develop tests to qualitatively characterize their errors on these inputs. Using
code generation as a case study, we find that OpenAI's Codex errs predictably
based on how the input prompt is framed, adjusts outputs towards anchors, and
is biased towards outputs that mimic frequent training examples. We then use
our framework to uncover high-impact errors such as incorrectly deleting files.
Our experiments suggest that cognitive science can be a useful jumping-off
point to better understand how contemporary machine learning systems behave.
