Combining Optimal Control and Learning for Visual Navigation in Novel
  Environments
  Model-based control is a popular paradigm for robot navigation because it can
leverage a known dynamics model to efficiently plan robust robot trajectories.
However, it is challenging to use model-based methods in settings where the
environment is a priori unknown and can only be observed partially through
on-board sensors on the robot. In this work, we address this short-coming by
coupling model-based control with learning-based perception. The learning-based
perception module produces a series of waypoints that guide the robot to the
goal via a collision-free path. These waypoints are used by a model-based
planner to generate a smooth and dynamically feasible trajectory that is
executed on the physical system using feedback control. Our experiments in
simulated real-world cluttered environments and on an actual ground vehicle
demonstrate that the proposed approach can reach goal locations more reliably
and efficiently in novel environments as compared to purely geometric
mapping-based or end-to-end learning-based alternatives. Our approach does not
rely on detailed explicit 3D maps of the environment, works well with low frame
rates, and generalizes well from simulation to the real world. Videos
describing our approach and experiments are available on the project website.
