Coagent Networks Revisited
  This work is aiming to discuss and close some of the gaps in the literature
on models using options (and more generally coagents). Briefly surveying the
theory behind these models, it also aims to provide a unifying point of view on
the many diverse examples that fall under a same category called coagent
network. Motivated by the result of [10] on parameter sharing of options, we
revisit the theory of (a)synchronous Coagent Network [8] by generalizing the
result to the context where parameters are shared among the function
approximators of coagents. The proof is more intuitive and uses the concept of
execution paths in a coagent network. Theoretically, this informs us of some
necessary modifications to the algorithms found in the literature which make
them more mathematically accurate. It also allows us to introduce a new simple
option framework, Feedforward Option Network, which outperforms the previous
option models in time to convergence and stability in the famous nonstationary
Four Rooms task. In addition, a stabilization effect is observed in
hierarchical models which justify the unnecessity of the target network in
training such models. Finally, we publish our code which allows us to be
flexible in our experiments settings.
