Expecting the Unexpected: Developing Autonomous-System Design Principles
  for Reacting to Unpredicted Events and Conditions
  When developing autonomous systems, engineers and other stakeholders make
great effort to prepare the system for all foreseeable events and conditions.
However, these systems are still bound to encounter events and conditions that
were not considered at design time. For reasons like safety, cost, or ethics,
it is often highly desired that these new situations be handled correctly upon
first encounter. In this paper we first justify our position that there will
always exist unpredicted events and conditions, driven among others by: new
inventions in the real world; the diversity of world-wide system deployments
and uses; and, the non-negligible probability that multiple seemingly unlikely
events, which may be neglected at design time, will not only occur, but occur
together. We then argue that despite this unpredictability property, handling
these events and conditions is indeed possible. Hence, we offer and exemplify
design principles that when applied in advance, can enable systems to deal, in
the future, with unpredicted circumstances. We conclude with a discussion of
how this work and a broader theoretical study of the unexpected can contribute
toward a foundation of engineering principles for developing trustworthy
next-generation autonomous systems.
