Offline Contextual Bandits with Overparameterized Models
  Recent results in supervised learning suggest that while overparameterized
models have the capacity to overfit, they in fact generalize quite well. We ask
whether the same phenomenon occurs for offline contextual bandits. Our results
are mixed. Value-based algorithms benefit from the same generalization behavior
as overparameterized supervised learning, but policy-based algorithms do not.
We show that this discrepancy is due to the \emph{action-stability} of their
objectives. An objective is action-stable if there exists a prediction
(action-value vector or action distribution) which is optimal no matter which
action is observed. While value-based objectives are action-stable,
policy-based objectives are unstable. We formally prove upper bounds on the
regret of overparameterized value-based learning and lower bounds on the regret
for policy-based algorithms. In our experiments with large neural networks,
this gap between action-stable value-based objectives and unstable policy-based
objectives leads to significant performance differences.
