Adversarial Active Exploration for Inverse Dynamics Model Learning
  We present an adversarial active exploration for inverse dynamics model
learning, a simple yet effective learning scheme that incentivizes exploration
in an environment without any human intervention. Our framework consists of a
deep reinforcement learning (DRL) agent and an inverse dynamics model
contesting with each other. The former collects training samples for the
latter, with an objective to maximize the error of the latter. The latter is
trained with samples collected by the former, and generates rewards for the
former when it fails to predict the actual action taken by the former. In such
a competitive setting, the DRL agent learns to generate samples that the
inverse dynamics model fails to predict correctly, while the inverse dynamics
model learns to adapt to the challenging samples. We further propose a reward
structure that ensures the DRL agent to collect only moderately hard samples
but not overly hard ones that prevent the inverse model from predicting
effectively. We evaluate the effectiveness of our method on several robotic arm
and hand manipulation tasks against multiple baseline models. Experimental
results show that our method is comparable to those directly trained with
expert demonstrations, and superior to the other baselines even without any
human priors.
