Learning Visual Servoing with Deep Features and Fitted Q-Iteration
  Visual servoing involves choosing actions that move a robot in response to
observations from a camera, in order to reach a goal configuration in the
world. Standard visual servoing approaches typically rely on manually designed
features and analytical dynamics models, which limits their generalization
capability and often requires extensive application-specific feature and model
engineering. In this work, we study how learned visual features, learned
predictive dynamics models, and reinforcement learning can be combined to learn
visual servoing mechanisms. We focus on target following, with the goal of
designing algorithms that can learn a visual servo using low amounts of data of
the target in question, to enable quick adaptation to new targets. Our approach
is based on servoing the camera in the space of learned visual features, rather
than image pixels or manually-designed keypoints. We demonstrate that standard
deep features, in our case taken from a model trained for object
classification, can be used together with a bilinear predictive model to learn
an effective visual servo that is robust to visual variation, changes in
viewing angle and appearance, and occlusions. A key component of our approach
is to use a sample-efficient fitted Q-iteration algorithm to learn which
features are best suited for the task at hand. We show that we can learn an
effective visual servo on a complex synthetic car following benchmark using
just 20 training trajectory samples for reinforcement learning. We demonstrate
substantial improvement over a conventional approach based on image pixels or
hand-designed keypoints, and we show an improvement in sample-efficiency of
more than two orders of magnitude over standard model-free deep reinforcement
learning algorithms. Videos are available at
http://rll.berkeley.edu/visual_servoing .
