Confidence-Aware Imitation Learning from Demonstrations with Varying
  Optimality
  Most existing imitation learning approaches assume the demonstrations are
drawn from experts who are optimal, but relaxing this assumption enables us to
use a wider range of data. Standard imitation learning may learn a suboptimal
policy from demonstrations with varying optimality. Prior works use confidence
scores or rankings to capture beneficial information from demonstrations with
varying optimality, but they suffer from many limitations, e.g., manually
annotated confidence scores or high average optimality of demonstrations. In
this paper, we propose a general framework to learn from demonstrations with
varying optimality that jointly learns the confidence score and a
well-performing policy. Our approach, Confidence-Aware Imitation Learning
(CAIL) learns a well-performing policy from confidence-reweighted
demonstrations, while using an outer loss to track the performance of our model
and to learn the confidence. We provide theoretical guarantees on the
convergence of CAIL and evaluate its performance in both simulated and real
robot experiments. Our results show that CAIL significantly outperforms other
imitation learning methods from demonstrations with varying optimality. We
further show that even without access to any optimal demonstrations, CAIL can
still learn a successful policy, and outperforms prior work.
