Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous
  Demonstration
  Learning from Demonstration (LfD) approaches empower end-users to teach
robots novel tasks via demonstrations of the desired behaviors, democratizing
access to robotics. A key challenge in LfD research is that users tend to
provide heterogeneous demonstrations for the same task due to various
strategies and preferences. Therefore, it is essential to develop LfD
algorithms that ensure \textit{flexibility} (the robot adapts to personalized
strategies), \textit{efficiency} (the robot achieves sample-efficient
adaptation), and \textit{scalability} (robot reuses a concise set of strategies
to represent a large amount of behaviors). In this paper, we propose a novel
algorithm, Dynamic Multi-Strategy Reward Distillation (DMSRD), which distills
common knowledge between heterogeneous demonstrations, leverages learned
strategies to construct mixture policies, and continues to improve by learning
from all available data. Our personalized, federated, and lifelong LfD
architecture surpasses benchmarks in two continuous control problems with an
average 77\% improvement in policy returns and 42\% improvement in log
likelihood, alongside stronger task reward correlation and more precise
strategy rewards.
