Scalable agent alignment via reward modeling: a research direction
  One obstacle to applying reinforcement learning algorithms to real-world
problems is the lack of suitable reward functions. Designing such reward
functions is difficult in part because the user only has an implicit
understanding of the task objective. This gives rise to the agent alignment
problem: how do we create agents that behave in accordance with the user's
intentions? We outline a high-level research direction to solve the agent
alignment problem centered around reward modeling: learning a reward function
from interaction with the user and optimizing the learned reward function with
reinforcement learning. We discuss the key challenges we expect to face when
scaling reward modeling to complex and general domains, concrete approaches to
mitigate these challenges, and ways to establish trust in the resulting agents.
