Preparing for the Unknown: Learning a Universal Policy with Online
  System Identification
  We present a new method of learning control policies that successfully
operate under unknown dynamic models. We create such policies by leveraging a
large number of training examples that are generated using a physical
simulator. Our system is made of two components: a Universal Policy (UP) and a
function for Online System Identification (OSI). We describe our control policy
as universal because it is trained over a wide array of dynamic models. These
variations in the dynamic model may include differences in mass and inertia of
the robots' components, variable friction coefficients, or unknown mass of an
object to be manipulated. By training the Universal Policy with this variation,
the control policy is prepared for a wider array of possible conditions when
executed in an unknown environment. The second part of our system uses the
recent state and action history of the system to predict the dynamics model
parameters mu. The value of mu from the Online System Identification is then
provided as input to the control policy (along with the system state).
Together, UP-OSI is a robust control policy that can be used across a wide
range of dynamic models, and that is also responsive to sudden changes in the
environment. We have evaluated the performance of this system on a variety of
tasks, including the problem of cart-pole swing-up, the double inverted
pendulum, locomotion of a hopper, and block-throwing of a manipulator. UP-OSI
is effective at these tasks across a wide range of dynamic models. Moreover,
when tested with dynamic models outside of the training range, UP-OSI
outperforms the Universal Policy alone, even when UP is given the actual value
of the model dynamics. In addition to the benefits of creating more robust
controllers, UP-OSI also holds out promise of narrowing the Reality Gap between
simulated and real physical systems.
