Modularity benefits reinforcement learning agents with competing
  homeostatic drives
  The problem of balancing conflicting needs is fundamental to intelligence.
Standard reinforcement learning algorithms maximize a scalar reward, which
requires combining different objective-specific rewards into a single number.
Alternatively, different objectives could also be combined at the level of
action value, such that specialist modules responsible for different objectives
submit different action suggestions to a decision process, each based on
rewards that are independent of one another. In this work, we explore the
potential benefits of this alternative strategy. We investigate a biologically
relevant multi-objective problem, the continual homeostasis of a set of
variables, and compare a monolithic deep Q-network to a modular network with a
dedicated Q-learner for each variable. We find that the modular agent: a)
requires minimal exogenously determined exploration; b) has improved sample
efficiency; and c) is more robust to out-of-domain perturbation.
