CAQL: Continuous Action Q-Learning
  Value-based reinforcement learning (RL) methods like Q-learning have shown
success in a variety of domains. One challenge in applying Q-learning to
continuous-action RL problems, however, is the continuous action maximization
(max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a
(class of) algorithm(s) for continuous-action Q-learning that can use several
plug-and-play optimizers for the max-Q problem. Leveraging recent optimization
results for deep neural networks, we show that max-Q can be solved optimally
using mixed-integer programming (MIP). When the Q-function representation has
sufficient power, MIP-based optimization gives rise to better policies and is
more robust than approximate methods (e.g., gradient ascent, cross-entropy
search). We further develop several techniques to accelerate inference in CAQL,
which despite their approximate nature, perform well. We compare CAQL with
state-of-the-art RL algorithms on benchmark continuous-control problems that
have different degrees of action constraints and show that CAQL outperforms
policy-based methods in heavily constrained environments, often dramatically.
