The Paradox of Choice: Using Attention in Hierarchical Reinforcement
  Learning
  Decision-making AI agents are often faced with two important challenges: the
depth of the planning horizon, and the branching factor due to having many
choices. Hierarchical reinforcement learning methods aim to solve the first
problem, by providing shortcuts that skip over multiple time steps. To cope
with the breadth, it is desirable to restrict the agent's attention at each
step to a reasonable number of possible choices. The concept of affordances
(Gibson, 1977) suggests that only certain actions are feasible in certain
states. In this work, we model "affordances" through an attention mechanism
that limits the available choices of temporally extended options. We present an
online, model-free algorithm to learn affordances that can be used to further
learn subgoal options. We investigate the role of hard versus soft attention in
training data collection, abstract value learning in long-horizon tasks, and
handling a growing number of choices. We identify and empirically illustrate
the settings in which the paradox of choice arises, i.e. when having fewer but
more meaningful choices improves the learning speed and performance of a
reinforcement learning agent.
