What Would Jiminy Cricket Do? Towards Agents That Behave Morally
  When making everyday decisions, people are guided by their conscience, an
internal sense of right and wrong. By contrast, artificial agents are currently
not endowed with a moral sense. As a consequence, they may learn to behave
immorally when trained on environments that ignore moral concerns, such as
violent video games. With the advent of generally capable agents that pretrain
on many environments, it will become necessary to mitigate inherited biases
from environments that teach immoral behavior. To facilitate the development of
agents that avoid causing wanton harm, we introduce Jiminy Cricket, an
environment suite of 25 text-based adventure games with thousands of diverse,
morally salient scenarios. By annotating every possible game state, the Jiminy
Cricket environments robustly evaluate whether agents can act morally while
maximizing reward. Using models with commonsense moral knowledge, we create an
elementary artificial conscience that assesses and guides agents. In extensive
experiments, we find that the artificial conscience approach can steer agents
towards moral behavior without sacrificing performance.
