Deep Reinforcement Learning with Modulated Hebbian plus Q Network
  Architecture
  This paper presents a new neural architecture that combines a modulated
Hebbian network (MOHN) with DQN, which we call modulated Hebbian plus Q network
architecture (MOHQA). The hypothesis is that such a combination allows MOHQA to
solve difficult partially observable Markov decision process (POMDP) problems
which impair temporal difference (TD)-based RL algorithms such as DQN, as the
TD error cannot be easily derived from observations. The key idea is to use a
Hebbian network with bio-inspired neural traces in order to bridge temporal
delays between actions and rewards when confounding observations and sparse
rewards result in inaccurate TD errors. In MOHQA, DQN learns low level features
and control, while the MOHN contributes to the high-level decisions by
associating rewards with past states and actions. Thus the proposed
architecture combines two modules with significantly different learning
algorithms, a Hebbian associative network and a classical DQN pipeline,
exploiting the advantages of both. Simulations on a set of POMDPs and on the
MALMO environment show that the proposed algorithm improved DQN's results and
even outperformed control tests with A2C, QRDQN+LSTM and REINFORCE algorithms
on some POMDPs with confounding stimuli and sparse rewards.
