Interference and Generalization in Temporal Difference Learning
  We study the link between generalization and interference in
temporal-difference (TD) learning. Interference is defined as the inner product
of two different gradients, representing their alignment. This quantity emerges
as being of interest from a variety of observations about neural networks,
parameter sharing and the dynamics of learning. We find that TD easily leads to
low-interference, under-generalizing parameters, while the effect seems
reversed in supervised learning. We hypothesize that the cause can be traced
back to the interplay between the dynamics of interference and bootstrapping.
This is supported empirically by several observations: the negative
relationship between the generalization gap and interference in TD, the
negative effect of bootstrapping on interference and the local coherence of
targets, and the contrast between the propagation rate of information in TD(0)
versus TD($\lambda$) and regression tasks such as Monte-Carlo policy
evaluation. We hope that these new findings can guide the future discovery of
better bootstrapping methods.
