Deep Reinforcement Learning with Feedback-based Exploration
  Deep Reinforcement Learning has enabled the control of increasingly complex
and high-dimensional problems. However, the need of vast amounts of data before
reasonable performance is attained prevents its widespread application. We
employ binary corrective feedback as a general and intuitive manner to
incorporate human intuition and domain knowledge in model-free machine
learning. The uncertainty in the policy and the corrective feedback is combined
directly in the action space as probabilistic conditional exploration. As a
result, the greatest part of the otherwise ignorant learning process can be
avoided. We demonstrate the proposed method, Predictive Probabilistic Merging
of Policies (PPMP), in combination with DDPG. In experiments on continuous
control problems of the OpenAI Gym, we achieve drastic improvements in sample
efficiency, final performance, and robustness to erroneous feedback, both for
human and synthetic feedback. Additionally, we show solutions beyond the
demonstrated knowledge.
