{'fraction_non_alphanumeric': 0.03538236542907015, 'fraction_numerical': 0.014884181723828844, 'mean_word_length': 4.807534246575343, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 4, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LazyDAgger: Reducing Context Switching in Interactive Imitation Learning

Corrective interventions while a robot is learning to automate a task provide an intuitive method for a human supervisor to assist the robot and convey information about desired behavior. However, these interventions can impose significant burden on a human supervisor, as each intervention interrupts other work the human is doing, incurs latency with each context switch between supervisor and autonomous control, and requires time to perform. We present LazyDAgger, which extends the interactive imitation learning (IL) algorithm SafeDAgger to reduce context switches between supervisor and autonomous control. We find that LazyDAgger improves the performance and robustness of the learned policy during both learning and execution while limiting burden on the supervisor. Simulation experiments suggest that LazyDAgger can reduce context switches by an average of 60% over SafeDAgger on 3 continuous control tasks while maintaining state-of-the-art policy performance. In physical fabric manipulation experiments with an ABB YuMi robot, LazyDAgger reduces context switches by 60% while achieving a 60% higher success rate than SafeDAgger at execution time.


I. INTRODUCTION
Imitation learning allows a robot to learn from human feedback and examples [3,4,33]. In particular, interactive imitation learning (IL) [24,45,53], in which a human supervisor periodically takes control of the robotic system during policy learning, has emerged as a popular imitation learning method, as interventions are a particularly intuitive form of human feedback [45]. However, a key challenge in interactive imitation learning is to reduce the burden that interventions place on the human supervisor [24,53].
One source of this burden is the cost of context switches between human and robot control. Context switches incur significant time cost, as a human must interrupt the task they are currently performing, acquire control of the robot, and gain sufficient situational awareness before beginning the intervention. As an illustrative example, consider a robot performing a task for which an action takes 1 time unit and an intervention requires two context switches (one at the start and one at the end). We define latency L as the number of time units associated with a single context switch. For instance, L 1 for a human supervisor who will need to pause an ongoing task and walk over to a robot that requires assistance. If the supervisor takes control 10 times for 2 actions each, she spends 20L + 20 time units helping the robot. In contrast, if the human takes control 2 times for 10 actions each, she spends only 4L + 20 time units. The latter significantly reduces the burden on the AUTOLAB at the University of California, Berkeley Correspondence to ryanhoque@berkeley.edu Fig. 1: LazyDAgger learns to cede control to a supervisor in states in which it estimates that its actions will significantly deviate from those of the supervisor. LazyDAgger reduces context switches between supervisor and autonomous control to reduce burden on a human supervisor working on multiple tasks.
supervisor. Furthermore, prior work suggests that frequent context switches can make it difficult for the supervisor to perform other tasks in parallel [46] or gain enough situational awareness to provide useful interventions [39].
We present LazyDAgger (Figure 1), an algorithm which initiates useful interventions while limiting context switches. The name LazyDAgger is inspired by the concept of lazy evaluation in programming language theory [29], where expressions are evaluated only when required to reduce computational burden. As in SafeDAgger [53], LazyDAgger learns a meta-controller which determines when to context switch based on the estimated discrepancy between the learner and supervisor. However, unlike SafeDAgger, LazyDAgger reduces context switching by (1) introducing asymmetric switching criteria and (2) injecting noise into the supervisor control actions to widen the distribution of visited states. One appealing property of this improved metacontroller is that even after training, LazyDAgger can be applied at execution time to improve the safety and reliability of autonomous policies with minimal context switching. We find that across 3 continuous control tasks in simulation, LazyDAgger achieves task performance on par with DAgger [40] with 88% fewer supervisor actions than DAgger and 60% fewer context switches than SafeDAgger. In physical fabric manipulation experiments, we observe similar results, and find that at execution time, LazyDAgger achieves 60% better task performance than SafeDAgger with 60% fewer context switches.

II. BACKGROUND AND RELATED WORK
Challenges in learning efficiency and reward function specification have inspired significant interest in algorithms that can leverage supervisor demonstrations and feedback for policy learning.
Learning from Offline Demonstrations: Learning from demonstrations [3,4,33] is a popular imitation learning approach, as it requires minimal supervisor burden: the supervisor provides a batch of offline demonstrations and gives no further input during policy learning. Many methods use demonstrations directly for policy learning [20,35,36,49], while others use reinforcement learning to train a policy using a reward function inferred from demonstrations [1,9,15,17,54]. Recent work has augmented demonstrations with additional offline information such as pairwise preferences [10,11], human gaze [42], and natural language descriptions [50]. While offline demonstrations are often simple to provide, the lack of online feedback makes it difficult to address specific bottlenecks in the learning process or errors in the resulting policy due to covariate shift [40].
Learning from Online Feedback: Many policy learning algorithms' poor performance stems from a lack of online supervisor guidance, motivating active learning methods such as DAgger, which queries the supervisor for an action in every state that the learner visits [40]. While DAgger has a number of desirable theoretical properties, labeling every state is costly in human time and can be a non-intuitive form of human feedback [27]. More generally, the idea of learning from action advice has been widely explored in imitation learning algorithms [5,21,22,28]. There has also been significant recent interest in active preference queries for learning reward functions from pairwise preferences over demonstrations [7,10,13,19,34,41]. However, many forms of human advice can be unintuitive, since the learner may visit states that are significantly far from those the human supervisor would visit, making it difficult for humans to judge what correct behavior looks like without interacting with the environment themselves [38,45].
Learning from Supervisor Interventions: There has been significant prior work on algorithms for learning policies from interventions. Kurenkov et al. [26] and Xie et al. [52] leverage interventions from suboptimal supervisors to accelerate policy learning, but assume that the supervisors are algorithmic and thus can be queried cheaply. Nolan Wagener [32], Saunders et al. [43], and Thananjeyan* et al. [47] also leverage interventions from algorithmic policies, but for constraint satisfaction during learning. Amir et al. [2], Kahn et al. [23], Kelly et al. [24], Mandlekar et al. [30], Spencer et al. [45], and Wang et al. [51] instead consider learning from human supervisors and present learning algorithms which utilize the timing and nature of human interventions to update the learned policy. By giving the human control for multiple timesteps in a row, these algorithms show improvements over methods that only hand over control on a state-by-state basis [6]. However, the above algorithms assume that the human is continuously monitoring the system to determine when to intervene, which may not be practical in large-scale systems or continuous learning settings [12,14,25,46]. Such algorithms also assume that the human knows when to cede control to the robot, which requires guessing how the robot will behave in the future. Zhang et al. [53] and Menda et al. [31] present imitation learning algorithms SafeDAgger and EnsembleDAgger, respectively, to address these issues by learning to request interventions from a supervisor based on measures such as state novelty or estimated discrepancy between the learner and supervisor actions. These methods can still be sample inefficient, and, as we discuss later, often result in significant context switching. By contrast, LazyDAgger encourages interventions that are both easier to provide and more informative. To do this, LazyDAgger prioritizes (1) sustained interventions, which allow the supervisor to act over a small number of contiguous sequences of states rather than a large number of disconnected intervals, and (2) interventions which demonstrate supervisor actions in novel states to increase robustness to covariate shift in the learned policy.

III. PROBLEM STATEMENT
We consider a setting in which a human supervisor is training a robot to reliably perform a task. The robot may query the human for assistance, upon which the supervisor takes control and teleoperates the robot until the system determines that it no longer needs assistance. We assume that the robot and human policy have the same action space, and that it is possible to pause task execution while waiting to transfer control. We formalize these ideas in the context of prior imitation learning literature.
We model the environment as a discrete-time Markov decision process (MDP) M with states s ∈ S, actions a ∈ A, and time horizon T [37]. The robot does not have access to the reward function or transition dynamics of M but can cede control to a human supervisor, who executes some deterministic policy π H : S → A. We refer to times when the robot is in control as autonomous mode and those in which the supervisor is in control as supervisor mode. We minimize a surrogate loss function J(π R ) to encourage the robot policy π R : S → A to match that of the supervisor (π H ): where L(π R (s), π H (s)) is an action discrepancy measure between π R (s) and π H (s) (e.g., the squared loss or 0-1 loss), and d π R t is the marginal state distribution at timestep t induced by executing π R in MDP M.
In interactive IL we require a meta-controller π that determines whether to query the robot policy π R or to query for an intervention from the human supervisor policy π H ; importantly, π consists of both (1) the high-level controller which decides whether to switch between π R and π H and (2) the low-level robot policy π R . A key objective in interactive IL is to minimize some notion of supervisor burden. To this end, let m I (s t ; π) be an indicator which records whether a context switch between autonomous (π R ) and supervisor (π H ) modes occurs at state s t (either direction). Then, we define C(π), the expected number of context switches in an episode under policy π, as follows: C(π) = ∑ T t=1 E s t ∼d π t [m I (s t ; π)], where d π t is the marginal state distribution at timestep t induced by executing the meta-controller π in MDP M. Similarly, let m H (s t ; π) indicate whether the system is in supervisor mode at state s t . We then define D(π), the expected number of supervisor actions in an episode for the policy π, as follows: . We define supervisor burden B(π) as the expected time cost imposed on the human supervisor. This can be expressed as the sum of the expected total number of time units spent in context switching and the expected total number of time units in which the supervisor is actually engaged in performing interventions: where L is context switch latency (Section I) in time units, and each time unit is the time it takes for the supervisor to execute a single action. The learning objective is to find a policy π that matches supervisor performance, π H , while limiting supervisor burden to lie within a threshold Γ b , set by the supervisor to an acceptable tolerance for a given task. To formalize this problem, we propose the following objective: where Π is the space of all meta-controllers, and π R is the low-level robot policy associated with meta-controller π .
IV. PRELIMINARIES: SAFEDAGGER We consider interactive IL in the context of the objective introduced in Equation (III.3): to maximize task reward while limiting supervisor burden. To do this, LazyDAgger builds on SafeDAgger [53], a state-of-the-art algorithm for interactive IL. SafeDAgger selects between autonomous mode and supervisor mode by training a binary action discrepancy classifier f to discriminate between "safe" states which have an action discrepancy below a threshold β H (i.e., states with L(π R (s), π H (s)) < β H ) and "unsafe" states (i.e. states with L(π R (s), π H (s)) ≥ β H ). The classifier f is a neural network with a sigmoid output layer (i.e., f (s) ∈ [0, 1]) that is trained to minimize binary cross-entropy (BCE) loss on the datapoints (s t , π H (s t )) sampled from a dataset D of trajectories collected from π H . This is written as follows: where the training labels are given by f * (π R (s t ), π H (s t )) = 1{L(π R (s t ), π H (s t )) ≥ β H }, and 1 denotes the indicator function. Thus, L S (π R (s t ), π H (s t ), f ) penalizes incorrectly classifying a "safe" state as "unsafe" and vice versa. SafeDAgger executes the meta-policy π which selects between π R and π H as follows:

Algorithm 1 LazyDAgger
Require: Number of epochs N, time steps per epoch T , intervention thresholds β H , β R , supervisor policy π H , noise σ 2 1: Collect D, D S offline with supervisor policy π H 2: if L(a t , a H t ) < β R then 13: Mode ← Autonomous 14: else 15: Mode ← Supervisor 16: else 17: Execute a t 18: where f (s t ) < 0.5 corresponds to a prediction that L(π R (s t ), π H (s t )) < β H , i.e., that s t is "safe." Intuitively, SafeDAgger only solicits supervisor actions when f predicts that the action discrepancy between π R and π H exceeds the safety threshold β H . Thus, SafeDAgger provides a mechanism for querying the supervisor for interventions only when necessary. In LazyDAgger, we utilize this same mechanism to query for interventions but enforce new properties once we enter these interventions to lengthen them and increase the diversity of states observed during the interventions.

V. LAZYDAGGER
We summarize LazyDAgger in Algorithm 1. In the initial phase (Lines 1-3), we train π R and safety classifier f on offline datasets collected from the supervisor policy π H . In the interactive learning phase (Lines 4-19), we evaluate and update the robot policy for N epochs, ceding control to the supervisor when the robot predicts a high action discrepancy.

A. Action Discrepancy Prediction
SafeDAgger uses the classifier f to select between π R and π H (Equation (IV.2)). However, in practice, this often leads to frequent context switching ( Figure 3). To mitigate this, we make two observations. First, we can leverage that in supervisor mode, we directly observe the supervisor's Fig. 2: LazyDAgger Switching Strategy: SafeDAgger switches between supervisor and autonomous mode if the predicted action discrepancy is above threshold β H . In contrast, LazyDAgger uses asymmetric switching criteria and switches to autonomous mode based on ground truth action discrepancy. The gap between β R and β H defines a hysteresis band [8]. actions. Thus, there is no need to use f , which may have approximation errors, to determine whether to remain in supervisor mode; instead, we can compute the ground-truth action discrepancy L(π R (s t ), π H (s t )) exactly for any state s t visited in supervisor mode by comparing the supplied supervisor action π H (s t ) with the action proposed by the robot policy π R (s t ). In contrast, SafeDAgger uses f to determine when to switch modes both in autonomous and supervisor mode, which can lead to very short interventions when f prematurely predicts that the agent can match the supervisor's actions. Second, to ensure the robot has returned to the supervisor's distribution, the robot should only switch back to autonomous mode when the action discrepancy falls below a threshold β R , where β R < β H . As illustrated in Figure 2, LazyDAgger's asymmetric switching criteria create a hysteresis band, as is often utilized in control theory [8]. Motivated by Eq. (III.3), we adjust β H to reduce context switches C(π) and adjust β R as a function of β H to increase intervention length. We hypothesize that redistributing the supervisor actions into fewer but longer sequences in this fashion both reduces burden on the supervisor and improves the quality of the online feedback for the robot. Details on setting these hyperparameter values in practice, the settings used in our experiments, and a hyperparameter sensitivity analysis are provided in the Appendix.

B. Noise Injection
If the safety classifier is querying for interventions at state s t , then the robot either does not have much experience in the neighborhood of s t or has trouble matching the demonstrations at s t . This motivates exploring novel states near s t so that the robot can receive maximal feedback on the correct behavior in areas of the state space where it predicts a large action discrepancy from the supervisor. Inspired by prior work that has identified noise injection as a useful tool for improving the performance of imitation learning algorithms (e.g. Laskey et al. [27] and Brown et al. [9]), we diversify the set of states visited in supervisor mode by injecting isotropic Gaussian noise into the supervisor's actions, where the variance σ 2 is a scalar hyperparameter (Line 11 in Algorithm 1).

VI. EXPERIMENTS
We study whether LazyDAgger can (1) reduce supervisor burden while (2) achieving similar or superior task performance compared to prior algorithms. Implementation details are provided in the supplementary material. In all experiments, L measures Euclidean distance.

A. Simulation Experiments: MuJoCo Benchmarks
Environments: We evaluate LazyDAgger and baselines on 3 continuous control environments from MuJoCo [48], a standard simulator for evaluating imitation and reinforcement learning algorithms. In particular, we evaluate on HalfCheetah-v2, Walker2D-v2 and Ant-v2.
Metrics: For LazyDAgger and all baselines, we report learning curves which indicate how quickly they can make task progress in addition to metrics regarding the burden imposed on the supervisor. To study supervisor burden, we report the number of supervisor actions, the number of context switches, and the total supervisor burden (as defined in Eq. (III.2)). Additionally, we define L * ≥ 0 to be the latency value such that for all L > L * , LazyDAgger has a lower supervisor burden than SafeDAgger. We report this We study task performance measured by final fabric coverage (A), total supervisor actions (B), and total context switches (C) for LazyDAgger and baselines in the Gym-Cloth environment from [44]. The horizontal dotted line shows the success threshold for fabric smoothing. LazyDAgger achieves higher final coverage than Behavior Cloning and SafeDAgger with fewer context switches than SafeDAgger but more supervisor actions. At execution time, we again observe that LazyDAgger achieves similar coverage as SafeDAgger but with fewer context switches. L * value, which we refer to as the cutoff latency, for all experiments to precisely study the types of domains in which LazyDAgger is most applicable.
Baselines: We compare LazyDAgger to Behavior Cloning [49], DAgger [40], and SafeDAgger [53] in terms of the total supervisor burden and task performance. The Behavior Cloning and DAgger comparisons evaluate the utility of human interventions, while the comparison to SafeDAgger, another interactive IL algorithm, evaluates the impact of soliciting fewer but longer interventions.
Experimental Setup: For all MuJoCo environments, we use a reinforcement learning agent trained with TD3 [16] as an algorithmic supervisor. We begin all LazyDAgger, SafeDAgger, and DAgger experiments by pre-training the robot policy with Behavior Cloning on 4,000 state-action pairs for 5 epochs, and similarly report results for Behavior Cloning after the 5th epoch. To ensure a fair comparison, Behavior Cloning uses additional offline data equal to the average amount of online data seen by LazyDAgger during training. All results are averaged over 3 random seeds.
Results: In Figure 3, we study the performance of LazyDAgger and baselines. After every epoch of training, we run the policy for 10 test rollouts where interventions are not allowed and report the task reward on these rollouts in Figure 3. Results suggest that LazyDAgger is able to match or outperform all baselines in terms of task performance across all simulation environments ( Figure 3A). Additionally, LazyDAgger requires far fewer context switches compared to SafeDAgger ( Figure 3D), while requesting a similar number of supervisor actions across domains ( Figure 3C): we observe a 79%, 56%, and 46% reduction in context switches on the HalfCheetah, Walker2D, and Ant environments respectively. LazyDAgger and SafeDAgger both use an order of magnitude fewer supervisor actions than DAgger. While SafeDAgger requests much fewer supervisor actions than LazyDAgger in the Ant environment, this limited amount of supervision is insufficient to match the task performance of LazyDAgger or any of the baselines, suggesting that SafeDAgger may be terminating interventions prematurely. We study the total supervisor burden of SafeDAgger and LazyDAgger as defined in Equation (III.2) and find that in HalfCheetah, Walker2D, and Ant, the cutoff latencies L * are 0.0, 4.3, and 7.6 respectively, i.e. LazyDAgger achieves lower supervisor burden in the HalfCheetah domain for any L as well as lower burden in Walker2D and Ant for L > 4.3 and L > 7.6 respectively. The results suggest that LazyDAgger can reduce total supervisor burden compared to SafeDAgger even for modest latency values, but that SafeDAgger may be a better option for settings with extremely low latency.
Ablations: We study 2 key ablations for LazyDAgger in simulation: (1) returning to autonomous mode with f (·) rather than using the ground truth discrepancy (LazyDAgger (-Switch to Auto) in Figure 3), and (2) removal of noise injection (LazyDAgger (-Noise)). LazyDAgger outperforms both ablations on all tasks, with the exception of ablation 1 on Walker2D, which performed similarly well. We also observe that LazyDAgger consistently requests more supervisor actions than either ablation. This aligns with the intuition that both using the ground truth action discrepancy to switch back to autonomous mode and injecting noise result in longer but more useful interventions that improve performance.

B. Fabric Smoothing in Simulation
Environment: We evaluate LazyDAgger on the fabric smoothing task from [44] (shown in Figure 4) using the simulation environment from [44]. The task requires smoothing an initially crumpled fabric and is challenging due to the infinite-dimensional state space and complex dynamics, motivating learning from human feedback. As in prior work [44], we utilize top-down 100 × 100 × 3 RGB image observations of the workspace and use actions which consist of a 2D pick point and a 2D pull vector. See [44] for further details on the fabric simulator.
Experimental Setup: We train a fabric smoothing policy in simulation using DAgger under supervision from an analytic corner-pulling policy that leverages the simulator's state to identify fabric corners, iterate through them, and pull them towards the corners of the workspace [44]. We transfer the resulting policy for a 16×16 grid of fabric into a new simulation environment with altered fabric dynamics (i.e. lower spring constant, altered fabric colors, and a higherfidelity 25×25 discretization) and evaluate LazyDAgger and baselines on how rapidly they can adapt the initial policy to the new domain. As in [44], we terminate rollouts when we exceed 10 time steps, 92% coverage, or have moved the fabric more than 20% out of bounds. We evaluate performance based on a coverage metric, which measures the percentage of the background plane that the fabric covers (fully smooth corresponds to a coverage of 100).

Results:
We report results for the fabric smoothing simulation experiments in Figure 4. Figure 4 (A) shows the performance of the SafeDAgger and LazyDAgger policies during learning. To generate this plot we periodically evaluated each policy on test rollouts without interventions. Figure 4 (B) and (C) show the number of supervisor actions and context switches required during learning; LazyDAgger performs fewer context switches than SafeDAgger but requires more supervisor actions as the interventions are longer. Results suggest that the cutoff latency (as defined in Section VI-A) is L * = 1.5 for fabric smoothing. Despite fewer context switches, LazyDAgger achieves comparable performance to SafeDAgger, suggesting that LazyDAgger can learn complex, high-dimensional robotic control policies while reducing the number of hand-offs to a supervisor. We also evaluate LazyDAgger-Execution and SafeDAgger-Execution, in which interventions are allowed but the policy is no longer updated (see Section VI-C). We see that in this case, LazyDAgger achieves similar final coverage as SafeDAgger with significantly fewer context switches.

C. Physical Fabric Manipulation Experiments
Environment: In physical experiments, we evaluate on a multi-stage fabric manipulation task with an ABB YuMi robot and a human supervisor ( Figure 5). Starting from a crumpled initial fabric state, the task consists of 3 stages: (1) fully smooth the fabric, (2) align the fabric corners with a tight crop of the workspace, and (3) fold the fabric into a triangular fold. Stage (2) in particular requires high precision, motivating human interventions. As in the fabric simulation experiments, we use top-down 100×100×3 RGB image observations of the workspace and have 4D actions consisting of a pick point and pull vector. The actions are converted to workspace coordinates with a standard calibration procedure and analytically mapped to the nearest point on the fabric. Human supervisor actions are provided through a point-andclick interface for specifying pick-and-place actions. See the supplement for further details.
Experimental Setup: Here we study how interventions can be leveraged to improve the final task performance even at execution time, in which policies are no longer being updated. We collect 20 offline task demonstrations and train an initial policy with behavior cloning. To prevent overfitting to a small amount of real data, we use standard data augmentation techniques such as rotating, scaling, changing brightness, and adding noise to create 10 times as many training examples. We then evaluate the behavior cloning agent (Behavior Cloning) and agents which use the SafeDAgger and LazyDAgger intervention criteria but do not update the policy with new experience or inject noise (SafeDAgger-Execution and LazyDAgger-Execution respectively). We terminate rollouts if the fabric has successfully reached the goal state of the final stage (i.e. forms a perfect or near-perfect dark brown right triangle as in Hoque et al. [18]; see Figure 5), more than 50% of the fabric mask is out of view in the current observation, the predicted pick point misses the fabric mask by approximately 50% of the plane or more, or 15 total actions have been executed (either autonomous or supervisor).
Results: We perform 10 physical trials of each technique. In Table I, we report both the overall task success rate and success rates for each of the three stages of the task: (1) Smoothing, (2) Alignment, and (3) Folding. We also report the total number of context switches, supervisor actions, and autonomous robot actions summed across all 10 trials for each algorithm (Behavior Cloning, SafeDAgger-Execution, LazyDAgger-Execution). In Figure 5 we provide representative rollouts for each algorithm. Results suggest that Behavior Cloning is insufficient for successfully completing the alignment stage with the required level of precision. SafeDAgger-Execution does not improve the task success rate significantly due to its inability to collect interventions long enough to navigate bottleneck regions in the task ( Figure 5). LazyDAgger-Execution, however, achieves a much higher success rate than SafeDAgger-Execution and Behavior Cloning with far fewer context switches than SafeDAgger-Execution: LazyDAgger-Execution requests 2.1 context switches on average per trial (i.e. 1.05 interventions) as opposed to 5.3 switches (i.e. 2.65 interventions). LazyDAgger-Execution trials also make far more task progress than the baselines, as all 10 trials reach the folding stage. LazyDAgger-Execution does request more supervisor actions than SafeDAgger-Execution, as in the simulation environments. LazyDAgger-Execution also requests more supervisor actions relative to the total amount of actions due to the more conservative switching criteria and the fact that successful episodes are shorter than unsuccessful episodes on average. Nevertheless, results suggest that for this task, LazyDAgger-Execution reduces supervisor burden for any L > L * = 0.28, a very low cutoff latency that includes all settings in which a context switch is at least as timeconsuming as an individual action (i.e. L ≥ 1).
In experiments, we find that SafeDAgger-Execution's short interventions lead to many instances of Failure Mode A (see Table I), as the policy is making task progress, but not quickly enough to perform the task. We observe that Failure Mode C is often due to the fabric reaching a highly irregular configuration that is not within the training data distribution, making it difficult for the robot policy to make progress. We find that SafeDAgger and LazyDAgger experience Failure Mode D at a similar rate as they use the same criteria to solicit interventions (but different termination criteria). However, we find that all of LazyDAgger's failures are due to Failure Mode D, while SafeDAgger also fails in Mode A due to premature termination of interventions.

VII. DISCUSSION AND FUTURE WORK
We propose context switching between robot and human control as a metric for supervisor burden in interactive imitation learning and present LazyDAgger, an algorithm which can be used to efficiently learn tasks while reducing this switching. We evaluate LazyDAgger on 3 continuous control benchmark environments in MuJoCo, a fabric smoothing environment in simulation, and a fabric manipulation task with an ABB YuMi robot and find that LazyDAgger is able to improve task performance while reducing context switching between the learner and robot by up to 79% over SafeDAgger. In subsequent work, we investigate more intervention criteria and apply robot-gated interventions to controlling a fleet of robots, where context switching can negatively impact task throughput.

VIII. ACKNOWLEDGMENTS
This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, and the CITRIS "People and Robots" (CPAR) Initiative. This research was supported in part by the Scalable Collaborative Human-Robot Learning (SCHooL) Project, NSF National Robotics Initiative Award 1734633. The authors were supported in part by donations from Google, Siemens, Toyota Research Institute, Autodesk, Honda, Intel, and Hewlett-Packard and by equipment grants from PhotoNeo, Nvidia, and Intuitive Surgical. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors. We thank colleagues Lawrence Chen, Jennifer Grannen, and Vincent Lim for providing helpful feedback and suggestions.

IX. APPENDIX
Here we provide further details on our MuJoCo experiments, hyperparameter sensitivity, simulated fabric experiments, and physical fabric experiments.

A. MuJoCo
As stated in the main text, we evaluate on the HalfCheetah-v2, Walker2D-v2, and Ant-v2 environments. To train the algorithmic supervisor, we utilize the TD3 implementation from OpenAI SpinningUp (https://spinningup. openai.com/en/latest/) with default hyperparameters and run for 100, 200, and 500 epochs respectively. The expert policies obtain rewards of 5330.78 ± 117.65, 3492.08 ± 1110.31, and 4492.88 ± 1580.42, respectively. Note that the experts for Walker2D and Ant have high variance, resulting in higher variance for the corresponding learning curves in Figure 3. We provide the state space dimensionality |S|, action space dimensionality |A|, and LazyDAgger hyperparameters (see Algorithm 1) for each environment in Table II. The β H value in the table is multiplied with the maximum possible action discrepancy ||a high − a low || 2 2 to become the threshold for training f (·). In MuJoCo environments, a high = 1 and a low = − 1. The β H value used for SafeDAgger in all experiments is chosen by the method provided in the paper introducing SafeDAgger [53]: the threshold at which roughly 20% of the initial offline dataset is classified as "unsafe." For LazyDAgger and all baselines, the actor policy π R (·) is a neural network with 2 hidden layers with 256 neurons each, rectified linear unit (ReLU) activation, and hyperbolic tangent output activation. For LazyDAgger and SafeDAgger, the discrepancy classifier f (·) is a neural network with 2 hidden layers with 128 neurons each, ReLU activation, and sigmoid output activation. We take 2,000 gradient steps per epoch and optimize with Adam and learning rate 1e-3 for both neural networks. To collect D and D S in Algorithm 1 and SafeDAgger, we randomly partition our dataset of 4,000 state-action pairs into 70% (2,800 state-action pairs) for D and 30% (1,200

B. LazyDAgger Switching Thresholds
As described in Section V-A, the main LazyDAgger hyperparameters are the safety thresholds for switching to supervisor control (β H ) and returning to autonomous control (β R ). To tune these hyperparameters in practice, we initialize β H and β R with the method in Zhang et al. [53]; again, this sets the safety threshold such that approximately 20% of the initial dataset is unsafe. We then tune β H higher to balance reducing the frequency of switching to the supervisor with Fig. 6: LazyDAgger β R and β H sensitivity heatmaps across the 3 MuJoCo environments. The x-axis denotes β H and the y-axis denotes β R . Note that β R is a function of β H . Each of the 3 environments was run 9 times with the different settings of β R and β H . As in Figure 3 we plot test reward, number of supervisor actions, and number of context switches. allowing enough supervision for high policy performance. Finally we set β R as a multiple of β H , starting from β R = β H and tuning downward to balance improving the performance and increasing intervention length with keeping the total number of actions moderate. Note that since these parameters are not automatically set, we must re-run experiments for each change of parameter values. Since this tuning results in unnecessary supervisor burden, eliminating or mitigating this requirement is an important direction for future work.
To analyze sensitivity to β R and β H , we plot the results of a grid search over parameter values on each of the MuJoCo environments in Figure 6. Note that a lighter color in the heatmap is more desirable for reward while a darker color is more desirable for actions and switches. We see that the supervisor burden in terms of actions and context switches is not very sensitive to the threshold as we increase β H but jumps significantly for the very low setting (β H = 5 × 10 −4 ) as a large amount of data points are classified as unsafe. Similarly, we see that reward is relatively stable (note the small heatmap range for HalfCheetah) as we decrease β H but can suffer for high values, as interventions are not requested frequently enough. Reward and supervisor burden are not as sensitive to β R but follow the same trends we expect, with higher reward and burden as β R decreases.

C. Fabric Smoothing in Simulation
1) Fabric Simulator: More information about the fabric simulator can be found in Seita et al. [44], but we review the salient details here. The fabric is modeled as a massspring system with a n × n square grid of point masses. Self-collision is implemented by applying a repulsive force between points that are sufficiently close together. Blender (https://blender.org/) is used to render the fabric in 100 × 100 × 3 RGB image observations. See Figure 4 for an example observation. The actions are 4D vectors consisting of a pick point (x, y) ∈ [−1, 1] 2 and a place point (∆x, ∆y) ∈ [−1, 1] 2 , where (x, y) = (−1, −1) corresponds to the bottom left corner of the plane while (∆x, ∆y) is multiplied by 2 to allow crossing the entire plane. In simulation, we initialize the fabric with coverage 41.1 ± 3.4% in the hardest (Tier 3) state distribution in [44] and end episodes if we exceed 10 time steps, cover at least 92% of the plane, are at least 20% out of bounds, or have exceeded a tearing threshold in one of the springs. We use the same algorithmic supervisor as [44], which repeatedly picks the coordinates of the corner furthest from its goal position and pulls toward this goal position. To facilitate transfer to the real world, we use the domain randomization techniques in [44] to vary the following parameters: The discrepancy classifier reuses the actor's convolutional layers by taking a forward pass through them. We do not backpropagate gradients through these layers when training the classifier, but rather fix these parameters after training the actor policy. The rest of the classifier network has three fully connected layers with the following parameters: 3) Training: Due to the large amount of data required to train fabric smoothing policies, we pretrain the actor policy (not the discrepancy classifier) in simulation. The learned policy is then fine-tuned to the new environment while the discrepancy classifier is trained from scratch. Since the algorithmic supervisor can be queried cheaply, we pretrain with DAgger as in [44]. To further accelerate training, we parallelize environment interaction across 20 CPUs, and before DAgger iterations we pretrain with 100 epochs of Behavior Cloning on the dataset of 20,232 stateaction pairs available at [44]'s project website. Additional training hyperparameters are given in Table III and the learning curve is given in Figure 7.

4) Experiments:
In sim-to-sim experiments, the initial policy is trained on a 16x16 grid of fabric in a range of colors centered around blue with a spring constant of k = 10, 000. We then adapt this policy to a new simulator with different physics parameters and an altered visual appearance. Specifically, in the new simulation environment, the fabric is a higher fidelity 25x25 grid with a lower spring constant of k = 2, 000 and a color of (R, G, B) = (204, 51, 204) (i.e. pink), which is outside the range of colors produced by domain randomization (Section IX-C.1). Hyperparameters are given in Table IV. D. Fabric Manipulation with the ABB YuMi 1) Experimental Setup: We manipulate a brown 10" by 10" square piece of fabric with a single parallel jaw gripper as shown in Figure 1. The gripper is equipped with reverse tweezers for more precise picking of deformable materials. Neural network architecture is consistent with Section IX-C.2 for both actor and safety classifier. We correct pick points that nearly miss the fabric by mapping to the nearest point on Initial dataset sizes and batch size are in terms of images after data augmentation, i.e. scaled up by a factor of 15 (see Section IX-D.4). Note that the offline data is split 70%/30% as in Section IX-A. the mask of the fabric, which we segment from the images by color. To convert neural network actions to robot grasps, we run a standard camera calibration procedure and perform top-down grasps at a fixed depth. By controlling the width of the tweezers via the applied force on the gripper, we can reliably pick only the top layer of the fabric at a given pick point. We provide LazyDAgger-Execution hyperparameters in Table V.
2) Image Processing Pipeline: In the simulator, the fabric is smoothed against a light background plane with the same size as the fully smoothed fabric (see Figure 4). Since the physical workspace is far larger than the fabric, we process each RGB image of the workspace by (1) taking a square crop, (2) rescaling to 100 × 100, and (3) denoising the image. Essentially we define a square crop of the workspace as the region to smooth and align against, and assume that the fabric starts in this region. These processed images are the observations that fill the replay buffer and are passed to the neural networks.
3) User Interface: When the system solicits human intervention, an interactive user interface displays a scaled-up version of the current observation. The human is able to click and drag on the image to provide a pick point and pull vector, respectively. The interface captures the input as pixel locations and analytically converts it to the action space of the environment (i.e. a ∈ [−1, 1] 4 ) for the robot to execute. See Figure 8 for a screen capture of the user interface. 4) Data Augmentation: To prevent overfitting to the small amount of real data, before adding each state-action pair to the replay buffer, we make 10 copies of it with the following data augmentation procedure, with transformations applied in a random order: • Change contrast to 85-115% of the original value.
• Change brightness to 90-110% of the original value.
• Change saturation to 95-105% of the original value.
• Add values uniformly between -10 and 10 to each channel of each pixel. • Apply a Gaussian blur with σ between 0 and 0.6. • Add Gaussian noise with σ between 0 and 3. • With probability 0.8, apply an affine transform that (1) scales each axis independently to 98-102% of its original size, (2) translates each axis independently by a value between -2% and 2%, and (3) rotates by a value between -5 and 5 degrees.