AI Ethics Statements -- Analysis and lessons learnt from NeurIPS Broader
  Impact Statements
  Ethics statements have been proposed as a mechanism to increase transparency
and promote reflection on the societal impacts of published research. In 2020,
the machine learning (ML) conference NeurIPS broke new ground by requiring that
all papers include a broader impact statement. This requirement was removed in
2021, in favour of a checklist approach. The 2020 statements therefore provide
a unique opportunity to learn from the broader impact experiment: to
investigate the benefits and challenges of this and similar governance
mechanisms, as well as providing an insight into how ML researchers think about
the societal impacts of their own work. Such learning is needed as NeurIPS and
other venues continue to question and adapt their policies. To enable this, we
have created a dataset containing the impact statements from all NeurIPS 2020
papers, along with additional information such as affiliation type, location
and subject area, and a simple visualisation tool for exploration. We also
provide an initial quantitative analysis of the dataset, covering
representation, engagement, common themes, and willingness to discuss potential
harms alongside benefits. We investigate how these vary by geography,
affiliation type and subject area. Drawing on these findings, we discuss the
potential benefits and negative outcomes of ethics statement requirements, and
their possible causes and associated challenges. These lead us to several
lessons to be learnt from the 2020 requirement: (i) the importance of creating
the right incentives, (ii) the need for clear expectations and guidance, and
(iii) the importance of transparency and constructive deliberation. We
encourage other researchers to use our dataset to provide additional analysis,
to further our understanding of how researchers responded to this requirement,
and to investigate the benefits and challenges of this and related mechanisms.
