DQN with model-based exploration: efficient learning on environments
  with sparse rewards
  We propose Deep Q-Networks (DQN) with model-based exploration, an algorithm
combining both model-free and model-based approaches that explores better and
learns environments with sparse rewards more efficiently. DQN is a
general-purpose, model-free algorithm and has been proven to perform well in a
variety of tasks including Atari 2600 games since it's first proposed by Minh
et el. However, like many other reinforcement learning (RL) algorithms, DQN
suffers from poor sample efficiency when rewards are sparse in an environment.
As a result, most of the transitions stored in the replay memory have no
informative reward signal, and provide limited value to the convergence and
training of the Q-Network. However, one insight is that these transitions can
be used to learn the dynamics of the environment as a supervised learning
problem. The transitions also provide information of the distribution of
visited states. Our algorithm utilizes these two observations to perform a
one-step planning during exploration to pick an action that leads to states
least likely to be seen, thus improving the performance of exploration. We
demonstrate our agent's performance in two classic environments with sparse
rewards in OpenAI gym: Mountain Car and Lunar Lander.
