Facial Feedback for Reinforcement Learning: A Case Study and Offline
  Analysis Using the TAMER Framework
  Interactive reinforcement learning provides a way for agents to learn to
solve tasks from evaluative feedback provided by a human user. Previous
research showed that humans give copious feedback early in training but very
sparsely thereafter. In this article, we investigate the potential of agent
learning from trainers' facial expressions via interpreting them as evaluative
feedback. To do so, we implemented TAMER which is a popular interactive
reinforcement learning method in a reinforcement-learning benchmark problem ---
Infinite Mario, and conducted the first large-scale study of TAMER involving
561 participants. With designed CNN-RNN model, our analysis shows that telling
trainers to use facial expressions and competition can improve the accuracies
for estimating positive and negative feedback using facial expressions. In
addition, our results with a simulation experiment show that learning solely
from predicted feedback based on facial expressions is possible and using
strong/effective prediction models or a regression method, facial responses
would significantly improve the performance of agents. Furthermore, our
experiment supports previous studies demonstrating the importance of
bi-directional feedback and competitive elements in the training interface.
