Jury Learning: Integrating Dissenting Voices into Machine Learning
  Models
  Whose labels should a machine learning (ML) algorithm learn to emulate? For
ML tasks ranging from online comment toxicity to misinformation detection to
medical diagnosis, different groups in society may have irreconcilable
disagreements about ground truth labels. Supervised ML today resolves these
label disagreements implicitly using majority vote, which overrides minority
groups' labels. We introduce jury learning, a supervised ML approach that
resolves these disagreements explicitly through the metaphor of a jury:
defining which people or groups, in what proportion, determine the classifier's
prediction. For example, a jury learning model for online toxicity might
centrally feature women and Black jurors, who are commonly targets of online
harassment. To enable jury learning, we contribute a deep learning architecture
that models every annotator in a dataset, samples from annotators' models to
populate the jury, then runs inference to classify. Our architecture enables
juries that dynamically adapt their composition, explore counterfactuals, and
visualize dissent.
