Receding Horizon Curiosity
  Sample-efficient exploration is crucial not only for discovering rewarding
experiences but also for adapting to environment changes in a task-agnostic
fashion. A principled treatment of the problem of optimal input synthesis for
system identification is provided within the framework of sequential Bayesian
experimental design. In this paper, we present an effective
trajectory-optimization-based approximate solution of this otherwise
intractable problem that models optimal exploration in an unknown Markov
decision process (MDP). By interleaving episodic exploration with Bayesian
nonlinear system identification, our algorithm takes advantage of the inductive
bias to explore in a directed manner, without assuming prior knowledge of the
MDP. Empirical evaluations indicate a clear advantage of the proposed algorithm
in terms of the rate of convergence and the final model fidelity when compared
to intrinsic-motivation-based algorithms employing exploration bonuses such as
prediction error and information gain. Moreover, our method maintains a
computational advantage over a recent model-based active exploration (MAX)
algorithm, by focusing on the information gain along trajectories instead of
seeking a global exploration policy. A reference implementation of our
algorithm and the conducted experiments is publicly available.
