System Safety and Artificial Intelligence
  This chapter formulates seven lessons for preventing harm in artificial
intelligence (AI) systems based on insights from the field of system safety for
software-based automation in safety-critical domains. New applications of AI
across societal domains and public organizations and infrastructures come with
new hazards, which lead to new forms of harm, both grave and pernicious. The
text addresses the lack of consensus for diagnosing and eliminating new AI
system hazards. For decades, the field of system safety has dealt with
accidents and harm in safety-critical systems governed by varying degrees of
software-based automation and decision-making. This field embraces the core
assumption of systems and control that AI systems cannot be safeguarded by
technical design choices on the model or algorithm alone, instead requiring an
end-to-end hazard analysis and design frame that includes the context of use,
impacted stakeholders and the formal and informal institutional environment in
which the system operates. Safety and other values are then inherently
socio-technical and emergent system properties that require design and control
measures to instantiate these across the technical, social and institutional
components of a system. This chapter honors system safety pioneer Nancy
Leveson, by situating her core lessons for today's AI system safety challenges.
For every lesson, concrete tools are offered for rethinking and reorganizing
the safety management of AI systems, both in design and governance. This
history tells us that effective AI safety management requires transdisciplinary
approaches and a shared language that allows involvement of all levels of
society.
