TAAC: Temporally Abstract Actor-Critic for Continuous Control
  We present temporally abstract actor-critic (TAAC), a simple but effective
off-policy RL algorithm that incorporates closed-loop temporal abstraction into
the actor-critic framework. TAAC adds a second-stage binary policy to choose
between the previous action and a new action output by an actor. Crucially, its
"act-or-repeat" decision hinges on the actually sampled action instead of the
expected behavior of the actor. This post-acting switching scheme let the
overall policy make more informed decisions. TAAC has two important features:
a) persistent exploration, and b) a new compare-through Q operator for
multi-step TD backup, specially tailored to the action repetition scenario. We
demonstrate TAAC's advantages over several strong baselines across 14
continuous control tasks. Our surprising finding reveals that while achieving
top performance, TAAC is able to "mine" a significant number of repeated
actions with the trained policy even on continuous tasks whose problem
structures on the surface seem to repel action repetition. This suggests that
aside from encouraging persistent exploration, action repetition can find its
place in a good policy behavior. Code is available at
https://github.com/hnyu/taac.
