Exploiting the Sign of the Advantage Function to Learn Deterministic
  Policies in Continuous Domains
  In the context of learning deterministic policies in continuous domains, we
revisit an approach, which was first proposed in Continuous Actor Critic
Learning Automaton (CACLA) and later extended in Neural Fitted Actor Critic
(NFAC). This approach is based on a policy update different from that of
deterministic policy gradient (DPG). Previous work has observed its excellent
performance empirically, but a theoretical justification is lacking. To fill
this gap, we provide a theoretical explanation to motivate this unorthodox
policy update by relating it to another update and making explicit the
objective function of the latter. We furthermore discuss in depth the
properties of these updates to get a deeper understanding of the overall
approach. In addition, we extend it and propose a new trust region algorithm,
Penalized NFAC (PeNFAC). Finally, we experimentally demonstrate in several
classic control problems that it surpasses the state-of-the-art algorithms to
learn deterministic policies.
