A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with
  Symbolic Reward Machines
  A misspecified reward can degrade sample efficiency and induce undesired
behaviors in reinforcement learning (RL) problems. We propose symbolic reward
machines for incorporating high-level task knowledge when specifying the reward
signals. Symbolic reward machines augment existing reward machine formalism by
allowing transitions to carry predicates and symbolic reward outputs. This
formalism lends itself well to inverse reinforcement learning, whereby the key
challenge is determining appropriate assignments to the symbolic values from a
few expert demonstrations. We propose a hierarchical Bayesian approach for
inferring the most likely assignments such that the concretized reward machine
can discriminate expert demonstrated trajectories from other trajectories with
high accuracy. Experimental results show that learned reward machines can
significantly improve training efficiency for complex RL tasks and generalize
well across different task environment configurations.
