Learning and Planning with a Semantic Model
  Building deep reinforcement learning agents that can generalize and adapt to
unseen environments remains a fundamental challenge for AI. This paper
describes progresses on this challenge in the context of man-made environments,
which are visually diverse but contain intrinsic semantic regularities. We
propose a hybrid model-based and model-free approach, LEArning and Planning
with Semantics (LEAPS), consisting of a multi-target sub-policy that acts on
visual inputs, and a Bayesian model over semantic structures. When placed in an
unseen environment, the agent plans with the semantic model to make high-level
decisions, proposes the next sub-target for the sub-policy to execute, and
updates the semantic model based on new observations. We perform experiments in
visual navigation tasks using House3D, a 3D environment that contains diverse
human-designed indoor scenes with real-world objects. LEAPS outperforms strong
baselines that do not explicitly plan using the semantic content.
