Deep Reinforcement Learning with Attention for Slate Markov Decision
  Processes with High-Dimensional States and Actions
  Many real-world problems come with action spaces represented as feature
vectors. Although high-dimensional control is a largely unsolved problem, there
has recently been progress for modest dimensionalities. Here we report on a
successful attempt at addressing problems of dimensionality as high as $2000$,
of a particular form. Motivated by important applications such as
recommendation systems that do not fit the standard reinforcement learning
frameworks, we introduce Slate Markov Decision Processes (slate-MDPs). A
Slate-MDP is an MDP with a combinatorial action space consisting of slates
(tuples) of primitive actions of which one is executed in an underlying MDP.
The agent does not control the choice of this executed action and the action
might not even be from the slate, e.g., for recommendation systems for which
all recommendations can be ignored. We use deep Q-learning based on feature
representations of both the state and action to learn the value of whole
slates. Unlike existing methods, we optimize for both the combinatorial and
sequential aspects of our tasks. The new agent's superiority over agents that
either ignore the combinatorial or sequential long-term value aspect is
demonstrated on a range of environments with dynamics from a real-world
recommendation system. Further, we use deep deterministic policy gradients to
learn a policy that for each position of the slate, guides attention towards
the part of the action space in which the value is the highest and we only
evaluate actions in this area. The attention is used within a sequentially
greedy procedure leveraging submodularity. Finally, we show how introducing
risk-seeking can dramatically improve the agents performance and ability to
discover more far reaching strategies.
