Resolving the Human Subjects Status of Machine Learning's Crowdworkers
  In recent years, machine learning (ML) has come to rely more heavily on
crowdworkers, both for building bigger datasets and for addressing research
questions requiring human interaction or judgment. Owing to the diverse tasks
performed by crowdworkers, and the myriad ways the resulting datasets are used,
it can be difficult to determine when these individuals are best thought of as
workers, versus as human subjects. These difficulties are compounded by
conflicting policies, with some institutions and researchers treating all ML
crowdwork as human subjects research, and other institutions holding that ML
crowdworkers rarely constitute human subjects. Additionally, few ML papers
involving crowdwork mention IRB oversight, raising the prospect that many might
not be in compliance with ethical and regulatory requirements. In this paper,
we focus on research in natural language processing to investigate the
appropriate designation of crowdsourcing studies and the unique challenges that
ML research poses for research oversight. Crucially, under the U.S. Common
Rule, these judgments hinge on determinations of "aboutness", both whom (or
what) the collected data is about and whom (or what) the analysis is about. We
highlight two challenges posed by ML: (1) the same set of workers can serve
multiple roles and provide many sorts of information; and (2) compared to the
life sciences and social sciences, ML research tends to embrace a dynamic
workflow, where research questions are seldom stated ex ante and data sharing
opens the door for future studies to ask questions about different targets from
the original study. In particular, our analysis exposes a potential loophole in
the Common Rule, where researchers can elude research ethics oversight by
splitting data collection and analysis into distinct studies. We offer several
policy recommendations to address these concerns.
