Accounting for Human Learning when Inferring Human Preferences
  Inverse reinforcement learning (IRL) is a common technique for inferring
human preferences from data. Standard IRL techniques tend to assume that the
human demonstrator is stationary, that is that their policy $\pi$ doesn't
change over time. In practice, humans interacting with a novel environment or
performing well on a novel task will change their demonstrations as they learn
more about the environment or task. We investigate the consequences of relaxing
this assumption of stationarity, in particular by modelling the human as
learning. Surprisingly, we find in some small examples that this can lead to
better inference than if the human was stationary. That is, by observing a
demonstrator who is themselves learning, a machine can infer more than by
observing a demonstrator who is noisily rational. In addition, we find evidence
that misspecification can lead to poor inference, suggesting that modelling
human learning is important, especially when the human is facing an unfamiliar
environment.
