Batch Recurrent Q-Learning for Backchannel Generation Towards Engaging
  Agents
  The ability to generate appropriate verbal and non-verbal backchannels by an
agent during human-robot interaction greatly enhances the interaction
experience. Backchannels are particularly important in applications like
tutoring and counseling, which require constant attention and engagement of the
user. We present here a method for training a robot for backchannel generation
during a human-robot interaction within the reinforcement learning (RL)
framework, with the goal of maintaining high engagement level. Since online
learning by interaction with a human is highly time-consuming and impractical,
we take advantage of the recorded human-to-human dataset and approach our
problem as a batch reinforcement learning problem. The dataset is utilized as a
batch data acquired by some behavior policy. We perform experiments with laughs
as a backchannel and train an agent with value-based techniques. In particular,
we demonstrate the effectiveness of recurrent layers in the approximate value
function for this problem, that boosts the performance in partially observable
environments. With off-policy policy evaluation, it is shown that the RL agents
are expected to produce more engagement than an agent trained from imitation
learning.
