Towards Sample-efficient Apprenticeship Learning from Suboptimal
  Demonstration
  Learning from Demonstration (LfD) seeks to democratize robotics by enabling
non-roboticist end-users to teach robots to perform novel tasks by providing
demonstrations. However, as demonstrators are typically non-experts, modern LfD
techniques are unable to produce policies much better than the suboptimal
demonstration. A previously-proposed framework, SSRR, has shown success in
learning from suboptimal demonstration but relies on noise-injected
trajectories to infer an idealized reward function. A random approach such as
noise-injection to generate trajectories has two key drawbacks: 1) Performance
degradation could be random depending on whether the noise is applied to vital
states and 2) Noise-injection generated trajectories may have limited
suboptimality and therefore will not accurately represent the whole scope of
suboptimality. We present Systematic Self-Supervised Reward Regression, S3RR,
to investigate systematic alternatives for trajectory degradation. We carry out
empirical evaluations and find S3RR can learn comparable or better reward
correlation with ground-truth against a state-of-the-art learning from
suboptimal demonstration framework.
