Robot Learning with Crash Constraints
  In the past decade, numerous machine learning algorithms have been shown to
successfully learn optimal policies to control real robotic systems. However,
it is common to encounter failing behaviors as the learning loop progresses.
Specifically, in robot applications where failing is undesired but not
catastrophic, many algorithms struggle with leveraging data obtained from
failures. This is usually caused by (i) the failed experiment ending
prematurely, or (ii) the acquired data being scarce or corrupted. Both
complicate the design of proper reward functions to penalize failures. In this
paper, we propose a framework that addresses those issues. We consider failing
behaviors as those that violate a constraint and address the problem of
learning with crash constraints, where no data is obtained upon constraint
violation. The no-data case is addressed by a novel GP model (GPCR) for the
constraint that combines discrete events (failure/success) with continuous
observations (only obtained upon success). We demonstrate the effectiveness of
our framework on simulated benchmarks and on a real jumping quadruped, where
the constraint threshold is unknown a priori. Experimental data is collected,
by means of constrained Bayesian optimization, directly on the real robot. Our
results outperform manual tuning and GPCR proves useful on estimating the
constraint threshold.
