Possibility Before Utility: Learning And Using Hierarchical Affordances
  Reinforcement learning algorithms struggle on tasks with complex hierarchical
dependency structures. Humans and other intelligent agents do not waste time
assessing the utility of every high-level action in existence, but instead only
consider ones they deem possible in the first place. By focusing only on what
is feasible, or "afforded", at the present moment, an agent can spend more time
both evaluating the utility of and acting on what matters. To this end, we
present Hierarchical Affordance Learning (HAL), a method that learns a model of
hierarchical affordances in order to prune impossible subtasks for more
effective learning. Existing works in hierarchical reinforcement learning
provide agents with structural representations of subtasks but are not
affordance-aware, and by grounding our definition of hierarchical affordances
in the present state, our approach is more flexible than the multitude of
approaches that ground their subtask dependencies in a symbolic history. While
these logic-based methods often require complete knowledge of the subtask
hierarchy, our approach is able to utilize incomplete and varying symbolic
specifications. Furthermore, we demonstrate that relative to
non-affordance-aware methods, HAL agents are better able to efficiently learn
complex tasks, navigate environment stochasticity, and acquire diverse skills
in the absence of extrinsic supervision -- all of which are hallmarks of human
learning.
