Dialogue Generation: From Imitation Learning to Inverse Reinforcement
  Learning
  The performance of adversarial dialogue generation models relies on the
quality of the reward signal produced by the discriminator. The reward signal
from a poor discriminator can be very sparse and unstable, which may lead the
generator to fall into a local optimum or to produce nonsense replies. To
alleviate the first problem, we first extend a recently proposed adversarial
dialogue generation method to an adversarial imitation learning solution. Then,
in the framework of adversarial inverse reinforcement learning, we propose a
new reward model for dialogue generation that can provide a more accurate and
precise reward signal for generator training. We evaluate the performance of
the resulting model with automatic metrics and human evaluations in two
annotation settings. Our experimental results demonstrate that our model can
generate more high-quality responses and achieve higher overall performance
than the state-of-the-art.
