EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline
  and Online RL
  Off-policy reinforcement learning holds the promise of sample-efficient
learning of decision-making policies by leveraging past experience. However, in
the offline RL setting -- where a fixed collection of interactions are provided
and no further interactions are allowed -- it has been shown that standard
off-policy RL methods can significantly underperform. Recently proposed methods
often aim to address this shortcoming by constraining learned policies to
remain close to the given dataset of interactions. In this work, we closely
investigate an important simplification of BCQ -- a prior approach for offline
RL -- which removes a heuristic design choice and naturally restricts extracted
policies to remain exactly within the support of a given behavior policy.
Importantly, in contrast to their original theoretical considerations, we
derive this simplified algorithm through the introduction of a novel backup
operator, Expected-Max Q-Learning (EMaQ), which is more closely related to the
resulting practical algorithm. Specifically, in addition to the distribution
support, EMaQ explicitly considers the number of samples and the proposal
distribution, allowing us to derive new sub-optimality bounds which can serve
as a novel measure of complexity for offline RL problems. In the offline RL
setting -- the main focus of this work -- EMaQ matches and outperforms prior
state-of-the-art in the D4RL benchmarks. In the online RL setting, we
demonstrate that EMaQ is competitive with Soft Actor Critic. The key
contributions of our empirical findings are demonstrating the importance of
careful generative model design for estimating behavior policies, and an
intuitive notion of complexity for offline RL problems. With its simple
interpretation and fewer moving parts, such as no explicit function
approximator representing the policy, EMaQ serves as a strong yet easy to
implement baseline for future work.
