Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement
  for Value Error
  In this work, we study the use of the Bellman equation as a surrogate
objective for value prediction accuracy. While the Bellman equation is uniquely
solved by the true value function over all state-action pairs, we find that the
Bellman error (the difference between both sides of the equation) is a poor
proxy for the accuracy of the value function. In particular, we show that (1)
due to cancellations from both sides of the Bellman equation, the magnitude of
the Bellman error is only weakly related to the distance to the true value
function, even when considering all state-action pairs, and (2) in the finite
data regime, the Bellman equation can be satisfied exactly by infinitely many
suboptimal solutions. This means that the Bellman error can be minimized
without improving the accuracy of the value function. We demonstrate these
phenomena through a series of propositions, illustrative toy examples, and
empirical analysis in standard benchmark domains.
