Heuristic-Based Weak Learning for Automated Decision-Making
  Machine learning systems impact many stakeholders and groups of users, often
disparately. Prior studies have reconciled conflicting user preferences by
aggregating a high volume of manually labeled pairwise comparisons, but this
technique may be costly or impractical. How can we lower the barrier to
participation in algorithm design? Instead of creating a simplified labeling
task for a crowd, we suggest collecting ranked decision-making heuristics from
a focused sample of affected users. With empirical data from two use cases, we
show that our weak learning approach, which requires little to no manual
labeling, agrees with participants' pairwise choices nearly as often as fully
supervised approaches.
