{'fraction_non_alphanumeric': 0.03827737719693556, 'fraction_numerical': 0.015096890491212258, 'mean_word_length': 4.785318559556787, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/


Introduction
Human supervision has been at the heart of the most significant recent advances in several domains such as computer vision [1][2][3][4] and natural language processing [5][6][7]. By intelligently extracting information from large-scale human-labeled datasets, autonomous machines have been able to reach near-or even super-human performance on decades-old problems such as image recognition and question answering. Roboticists have also attempted to tackle robot manipulation through learning from human datasets, using the paradigms of Imitation Learning [8][9][10] and Batch (Offline) Reinforcement Learning [11][12][13], where datasets consisting of robot arm trajectories, action labels at each timestep, and possibly reward labels, are used to train closed-loop policies.
As in other domains, large offline datasets offer several benefits such as scale, portability, and reproducible evaluations to measure progress. Recently, there has been considerable progress in offline learning for robot manipulation from human demonstrations [9,14,10]. Despite these advances, the offline learning paradigm has not been nearly as disruptive in robotics as in other disciplines -there is a large gap between autonomous robot manipulation capabilities and the wide range of tasks that humans can solve effortlessly using physical and cognitive intelligence. What has inhibited the use of large human-provided datasets to address this gap?
In contrast to other domains where supervised learning has been successful, robotic manipulation is a time-evolving dynamical system, requiring fine-grained real-time control to guide robot arms successfully through tasks -consequently, data collection can present technical challenges requiring arXiv:2108.03298v2 [cs.RO] 25 Sep 2021 We collect datasets across 6 operators of varying proficiency and evaluate offline policy learning methods on 8 challenging manipulation tasks that test a wide range of manipulation capabilities including pick-and-place, multi-arm coordination, and high-precision insertion and assembly.
specialized systems [15], which can explain why large-scale human-provided datasets [16,17] have not been very prevalent. Learning from such datasets can also present several challenges. Human demonstrations can differ from machine-generated datasets (a recent trend in benchmarks for offline policy learning [18,19]) due to a non-Markovian decision process, since humans may not act purely based on the current observation. There can also be significant variance in both data quality and solution strategy when collecting data from multiple humans [20]. Differences from classic supervised learning, such as a mismatch between training and evaluation objectives (task success rate), can make selecting a final policy challenging [21,22], especially in real-world settings where evaluating each policy on a robot can be infeasible. Finally, offline learning is sensitive to state and action space coverage (dataset size) and agent design decisions.
Studying these challenges in the context of robot manipulation and human-provided datasets could be a stepping stone to closing the gap between robot and human manipulation capabilities. Unfortunately, a lack of suitable benchmark and human datasets have made studying this setting difficult. Prior works are either limited to studying simple 2D environments [23] or using data generated from hard-coded policies [24,25]. In this paper, we address this need by presenting a study of data-driven offline policy learning methods on several human-provided robot manipulation datasets. We collect task demonstrations from human teleoperators across a broad range of simulated and real world manipulation tasks and investigate several factors that play a role in learning from such data.
From our results, we point out several lessons to guide future research in leveraging human supervision for robot manipulation effectively. We find that history-dependent models can be extremely effective in learning from single and multi-human datasets while state-of-the-art batch RL algorithms struggle to learn from such datasets, and that the choice of observation space and hyperparameters play a substantial role in training proficient policies. We also find that there is substantial promise for solving more complex tasks using large-scale human datasets and that our insights directly transfer to real-world scenarios, making this an important setting to explore further.

Challenges in Offline Learning from Human Datasets
In this section, we outline five challenges in offline learning from human datasets that motivate different factors that we investigate in our study.
(C1) Data from Non-Markovian Decision Process. Human demonstrations can differ substantially from machine-generated demonstrations because humans may not act purely based on a single current observation. External factors (teleoperation device, past actions, history of episode) may all play a role. Prior work [20] has noted substantial benefits from leveraging models that are history-dependent and / or with temporal abstraction to learn from human demonstrations. We investigate various design choices related to such architectures in this study.  14.0 ± 4.3 0.7 ± 0.9 60.7 ± 5.0 52.7 ± 5.0 Transport (MH) 11.3 ± 2. 5 65.3 ± 7.4 2.6 ± 0.9 0.0 ± 0.0 14.0 ± 1.6 10.7 ± 0.9 Table 1: Results on Low-Dimensional Observations. We present success rates averaged over 3 seeds for each method across the low-dim Machine-Generated (MG), Proficient-Human (PH), and Multi-Human (MH) datasets. The results show that methods that model temporal correlations (BC-RNN, HBC, IRIS) exhibit strong performance on human datasets. Furthermore, while Batch RL algorithms like BCQ are proficient on machine-generated data, they perform poorly on human datasets.
(C2) Variance in Demonstration Quality from Multiple Humans. Prior work [20,17] has found that data collected from several humans can differ substantially in both demonstration proficiency and solution strategy. Differences in supervisor proficiency can manifest in many ways, such as large variations in trajectory length and noise in robot movement or mistakes (e.g. missed grasps). In our study, we evaluate offline policy learning algorithms on such datasets. While recent batch RL algorithms have shown an excellent ability to learn from mixed quality machine-generated datasets [26,27], we empirically find that they fail to learn well from mixed quality human data.
(C3) Dependence on Dataset Size. Offline policy learning is sensitive to the state and action space coverage in the dataset, and by extension, the size of the dataset itself. In our study, we investigate how dataset sizes affect policy performance. This analysis is useful to understand the value of adding more data -an important consideration since collecting human demonstrations can be costly.
(C4) Mismatch between Training and Evaluation Objectives. Unlike traditional supervised learning, where model selection can be achieved by using the model with the lowest validation loss [21], offline policy learning often suffers from the fact that the training objective is only a surrogate for the true objective of interest (e.g. task success rate), and policy performance can change significantly from epoch to epoch. This makes it difficult to select the best trained model [19,28,29]. In our study, we evaluate each policy checkpoint online in the environment in simulation, and report the best policy success rate per training run. We use these ground-truth values to understand the effectiveness of different selection criteria, and confirm that offline policy selection is an important problem, especially in real-world scenarios where large-scale empirical evaluation is difficult.
(C5) High Sensitivity to Agent Design Decisions. Prior studies on machine-generated datasets have shown that offline policy learning can be extremely sensitive to hyperparameter choices [19,28]. In our study, we explore how agent design decisions affect policy performances, including the choice of agent architecture, agent observation space, and hyperparameter choices per algorithm. This results in several practical conclusions that should prove useful to researchers and practitioners alike. We further show that important design decisions made through our study in simulation directly translate to effective policy learning on real world tasks and datasets.

Tasks
We conducted our study across 5 simulated and 3 real world tasks. The tasks were chosen to test a broad range of manipulation capabilities. See Fig 1 and Appendix E for more details.
Lift (sim + real)). The robot arm must lift a small cube. This is the simplest task.
Can (sim + real). The robot must place a coke can from a large bin into a smaller target bin. Slightly more challenging than Lift, since picking the can is harder than picking the cube, and the can must also be placed into the bin.
Square (sim). The robot must pick a square nut and place it on a rod. Substantially more difficult than Lift and Pick Place Can due to the precision needed to pick up the nut and insert it on the rod.  Table 2: Results on Suboptimal Human Data. We present success rates averaged over 3 seeds for each method across different subsets of the Multi-Human datasets, corresponding to mixtures of demonstrations from "Better", "Adequate", and "Worse" human operators, and finally on a diagnostic dataset with paired success and failure human trajectories for each starting initialization. Results indicate that BC-RNN is a strong baseline, and that Batch RL methods perform poorly across all datasets, even on the simple diagnostic dataset.
Transport (sim). Two robot arms must transfer a hammer from a closed container on a shelf to a target bin on another shelf. One robot arm must retrieve the hammer from the container, while the other arm must clear the target bin by moving a piece of trash to the nearby receptacle. Finally, one arm must hand the hammer over to the other, which must place the hammer in the target bin.
Tool Hang (sim + real). A robot arm must assemble a frame consisting of a base piece and hook piece by inserting the hook into the base, and hang a wrench on the hook. This is the most difficult task due to the multiple stages that each require precise, and dexterous, rotation-heavy movements.

Data Collection
To study the effect of dataset source, we collected data from three different sources -Machine-Generated, Proficient-Human, and Multi-Human (more details in Appendix B).
Machine-Generated (MG). We collected these datasets by first training a state-of-the-art RL algorithm [30] on the Lift and Can task, taking agent checkpoints that are saved regularly during training, and collecting 300 rollout trajectories from each checkpoint. Consequently, these datasets are comprised of mixtures of expert and suboptimal data, and resemble datasets from common offline RL benchmarks [18,19]. We excluded other tasks because they could not be solved by the RL algorithm even with substantial tuning. See the appendix for more details.
Proficient-Human (PH) and Multi-Human (MH). Datasets are collected by humans through Robo-Turk [15,17], a remote teleoperation platform. The PH datasets consist of 200 demonstrations collected by a single, experienced teleoperator, while the MH datasets consist of 300 demonstrations, collected by 6 teleoperators of varying proficiency, each of which provided 50 demonstrations. The 6 teleoperators consisted of a "better" group of 2 experienced operators, an "okay" group of 2 adequate operators, and a "worse" group of 2 inexperienced operators. These data subsets in the Multi-Human data allowed us to investigate the ability of algorithms to deal with mixed quality human data.
Observation Modalities. To study the effect of observation modalities, we capture a diverse set of sensor streams when collecting the dataset, including end-effector, gripper fingers, and joints, groundtruth object poses, and images from an external camera and wrist-mounted camera per robot arm (see Appendix E). We have two observation spaces -"low-dim" and "image". Both include end-effector poses and gripper finger positions, and only differ in whether ground-truth object information is used (low-dim) or whether that information is replaced by the available camera observations (image).
in prior work to learn offline from teleoperated human demonstrations, while BCQ and CQL are commonly-used offline RL algorithms (see Appendix C). We use binary task completion rewards for all our experiments. Each agent is trained for N epochs, where each epoch consists of M gradient steps, and evaluated every E epochs, by running 50 rollouts in the environment and reporting the success rate over a maximum horizon. For each agent, we report the maximum success rate over the coarse of training, and average over 3 seeds.

Experiments
In this section, we present each factor that we explored in our study, and note the relevant challenges from Sec. 2 that each pertains to.

Algorithm Comparison on Single and Multi-Human Demonstrations (C1, C2)
We trained and evaluated all algorithms on the Proficient-Human (PH) and Multi-Human (MH) datasets and report the average success rates across 3 seeds in Table 1.
Observation history is crucial for good performance. There is a substantial performance gap between BC-RNN and BC, which highlights the benefits of history-dependence. The performance gap is larger for longer-horizon tasks (e.g. ∼ 55% for Transport (PH) compared to ∼ 5% for Square (PH)) and for multi-human data compared to single-human data (e.g. ∼ 25% for Square (MH) compared to ∼ 5% for Square (PH)). Interestingly, results are lower for MH datasets compared to PH datasets, even though the MH datasets contain 100 more demos (300 demos vs. 200 demos). This most likely stems from the presence of suboptimal and multimodal data in the MH datasets.
Batch RL algorithms perform poorly on Human Datasets. Recent batch (offline) RL algorithms such as BCQ and CQL have demonstrated excellent results in learning from suboptimal and multimodal agent-generated datasets. Our results confirm the capacity of such algorithms to work well -BCQ in particular performs strongly on our agent-generated MG datasets that consist of a diverse mixture of good and poor policies. Surprisingly though, neither BCQ nor CQL performs particularly well on these human-generated datasets. This puts the ability of such algorithms to learn from more   We compare how performance decreases when choosing the policy to evaluate by using the lowest validation loss, or when using the final trained checkpoint, with respect to the best policy performance. (right) Results on Image Observations. We present success rates for each method across the image observation human datasets. BC-RNN maintains nearly the same performance as learning from ground-truth observations, providing an optimistic view for learning with real-world raw sensory observations. natural dataset distributions into question (instead of those collected via RL exploration or pre-trained agents). There is an opportunity for future work in batch RL to resolve this gap.

Learning from Suboptimal Human Data (C2)
To further investigate how algorithms deal with suboptimal human data, we split our MH datasets into smaller subsets based on the proficiency of the human operators. The MH-Better, MH-Okay, and MH-Worse are the 100 demo subsets corresponding to the 2 "better", 2 "okay", and 2 "worse" operators respectively, while MH Worse-Okay, MH Worse-Better, and MH Okay-Better are the 200 demo subsets corresponding to the mixture of the previous subsets. Similar data mixtures have been used for evaluations in batch RL [18]. Appendix B shows the average trajectory lengths in each data subset -lower quality datasets contain demonstrations that take more time to solve the task.
BC-RNN is a strong baseline on suboptimal human data, but there is room for improvement. Diagnostic dataset shows that Batch RL struggles in simpler settings as well. The final row of Table 2 shows additional results on a diagnostic dataset termed Can-Paired, where a single operator collected 2 demonstrations for each of 100 task initializations -one successful demonstration, and one where the can is tossed outside of the bin (task failure), for a total of 200 demonstrations. There is a strong expectation for batch RL algorithms to be able to distinguish between actions leading to successful placement and actions leading to task failure, but even in this simple setting, most algorithms suffer, providing a pessimistic view of the state-of-the-art. The 5% improvement that IRIS provides over BC-RNN suggests that introducing history-dependence into state-of-the-art batch RL algorithms might be a promising direction for future work.

Effect of Observation Space (C5)
Learning from image observations can match low-dim agent performance. In Table 3, we present policy learning results when using image observations instead of ground-truth object locations -an important setting for real-world policy learning. BC-RNN still maintains superior performance improvements over BC on the complex Square and Transport tasks, and with the exception of Transport (MH), maintains nearly the same performance as learning from ground-truth observations. This result provides an optimistic view for learning with real-world raw sensory observations.
Features used for robot proprioception can matter. In Fig 2a, we study the effect of adding end effector velocities to the observations (+ EEF Vel), and joint positions and velocities to the observations (+ Joint). Surprisingly, we find that including end effector velocity information, and joint information hurts agents trained on low-dim observations substantially (49%-88% relative performance drop), while image-based agents are more tolerant to the inclusion of this extra information (2%-29% relative performance drop). We hypothesize that performance drops might be due to overfitting to the presence of this extra information not needed for solving these tasks. Thus, practitioners should take care to engineer the robot observation space and exclude possibly irrelevant informationinformation-hiding can be a powerful paradigm for training proficient robots [44].
Image randomization and wrist observations can be crucial for manipulation tasks. In Fig 2a, we report performance drops from removing pixel shift image randomization (-Rand) and the wrist camera (-Wrist) from image-based agents to understand their importance. We see that removing randomization results in 47% and 35% relative performance drops on Square and Transport respectively, and removing wrist images results in 9% and 43% relative drops. Consequently, both wrist camera images and image randomization play a substantial role in producing performant policies. We confirm the importance of each for visuomotor imitation in the real world as well (see Sec 4.7). Wrist observations likely help the robot improve gripper alignment for grasping and randomization helps the policy develop invariance for portions of the image that are not important for action prediction.

Effect of Hyperparameter Choice (C5)
In this section, we take our default hyperparameters for BC-RNN and study the effect of changing a subset of them to report practical recommendations for learning from human datasets (see Appendix I for BCQ and CQL). We present our results in (larger LR) Increasing the learning rate from 1e-4 to 1e-3 affects the performance of image-agents substantially (drop of 35%-63%), while low-dim agents are more tolerant to the change. (no GMM) Using a deterministic policy instead of learning a GMM action distribution results in significant relative performance drops on the MH datasets (especially low-dim Transport, with a drop of 58%).
(larger MLP) Using a larger MLP size at each RNN timestep reduces performance uniformly, suggesting that it is possible to overfit to dataset actions if network architectures are too large.
(shallow Conv) Using a shallow convolutional network [45] instead of the ResNet backbone [46] for encoding image observations reduces performance significantly -with relative drops of 25%-62%, suggesting that large-capacity visual encoders are crucial for visuomotor imitation. (smaller RNN dim) Reducing the size of the RNN hidden dimension from 400 to 100 (low-dim) and 1000 to 400 (image) uniformly decreases performance (drops of 3%-58%), showing the importance of a large RNN hidden dimension. (Recommendations) We recommend tuning the LR (especially for image agents) and network structure (MLP size, size of RNN dim) carefully. Opting to use a GMM policy and a ResNet encoder appears to be uniformly better.

Selecting a Policy to Evaluate (C4)
Model selection in offline policy learning can be challenging -for this reason, in our simulation experiments, we evaluated every policy checkpoint online and reported the best one. This is not feasible for real-world settings, making offline policy selection desirable. In Fig 4a, we show that this can be non-trivial, by showing the relative performance drop when selecting the policy using the best loss on validation data (common in supervised learning), and when using the final training checkpoint as well (common in offline RL [47,18,27]) -in both cases, the selected policy is significantly worse than the best one (10% to 100% decrease). See Appendix G for more detailed results and discussion. This motivates the need for better offline evaluation metrics.

Effect of Dataset Size (C3)
To study how dataset size impacts performance, we formed smaller 20% and 50% subsets of our human datasets by sampling trajectories. We evaluate low-dim and image BC-RNN agents across these subsets in Table 27 and Table 28. There are several promising results here. We first note that less complex tasks (Lift, Can) can yield proficient policies (75%-100% success rate) using a small fraction of the data (20%). Second, while policies trained on more complex tasks (Square, Transport) suffer substantially when using 50% or 20% of the data, the converse is also true -adding more data (e.g. moving from 20% to 50% or 50% to 100% size) can result in significant policy improvement. This confirms the value of using large human datasets as a means to obtain proficient policies for challenging and complex manipulation tasks.

Applicability to Real-World Settings
Here, we show that design decisions made in simulation can potentially transfer to real world settings. We collected 3 additional real-world datasets with a Franka robotic arm -Lift (Real), Can (Real), and Tool Hang (Real). Each consists of 200 trajectories collected by one operator. We train BC-RNN and report the final policy checkpoint success rate, over 30 rollouts, due to the time-consuming nature of real world policy evaluation. We also emphasize that no real-world hyperparameter tuning took place, so our results are a lower bound. We were able to train proficient Lift (96.7%) and Can (73.3%) policies, and the Tool Hang (3.3%) policy is able to generate some task successes, despite the extremely difficult nature of the task. Furthermore, as in Sec. 4.3, we validate the importance of pixel shift randomization and the wrist camera by ablating each component on the Can task, and show that including both is the difference between a proficient and non-proficient real-world policy -Can (-Rand) (26.7%), Can (-Wrist) (43.3%).

Discussion
In this section, we summarize the lessons from our study and make recommendations for future work.
(L1) Models with temporal abstraction can be extremely effective in learning from human datasets. In Sec 4.1 and Sec 4.2, we demonstrated that history-dependent models (BC-RNN, HBC, and IRIS) are particularly effective in learning from human datasets compared to algorithms that do not take temporal context into account.
(L2) Need to improve the ability of batch (offline) RL to learn from suboptimal human datasets. Sec 4.2 and Appendix I demonstrated that state-of-the-art batch RL algorithms are excellent at learning from suboptimal machine-generated datasets but much worse at learning from suboptimal human datasets. They even struggled with a diagnostic dataset with paired good and bad human demonstration trajectories while IRIS was able to improve slightly on BC-RNN, suggesting that combining history-dependence with value learning might be a good place to start for improving batch RL methods [48][49][50][51]. This also demonstrates a need to start benchmarking new batch RL algorithms on human datasets instead of purely on machine-generated datasets.
(L3) Improving offline policy selection is important for real world settings. Sec 4.5 demonstrated the need for better ways to select an evaluation policy in an offline manner. We hope that our datasets can help supplement other efforts [29].
(L4) Observation space plays a large role and hyperparameters matter. Sec 4.3 demonstrates that policies trained on low-dim observations can be very sensitive to the choice of robot proprioception, while pixel shift randomization and wrist camera images are critical for effective visuomotor policy learning. The choice of observation space for imitation merits careful consideration -other work has also confirmed the importance of feature representations used for offline policy learning [52,53]. Sec 4.4 and Appendix I made practical recommendations for choosing hyperparameters to learn from human data.
(L5) There is substantial promise for solving more complex tasks using large-scale human datasets. Sec 4.6 showed that adding more data can result in significant policy improvement on complex tasks. Table 3 and Sec 4.7 shows that we could learn proficient policies on the Tool Hang task, our most complex task, without any hyperparameter tuning on the task or dataset. Together, these results show the potential of large human datasets as a means to solve challenging and complex manipulation tasks.
(L6) Study results transfer to real-world settings. In Sec 4.7, we showed that we could directly apply hyperparameters that were tuned on simulated tasks directly to real-world datasets and tasks. This provides promise for using our tasks, datasets, and codebase to enable reproducible evaluation in simulation, while also being confident that conclusions can transfer to real-world settings.
Going forward, we hope that the datasets, tasks, code, and subsequent insights of our study will serve researchers and practitioners alike.
Offline Learning from Demonstrations: Imitation Learning (IL) is a popular paradigm for training policies from a set of demonstrations. Offline Imitation Learning typically consists of variants of Behavioral Cloning (BC) [8], where a policy is trained to output the same actions as the ones taken by the demonstrator in each state. Offline IL has been used extensively in robotic manipulation [31-34, 9, 10, 25, 35, 36]. Typically, IL assumes that the demonstration data is optimal. By contrast, Batch (Offline) Reinforcement Learning [11,12] is a method to learn from demonstrations that can consist of both good and bad quality data, by leveraging reward annotations in the datasets. Prior algorithms [26,27,[37][38][39][40][41][42][43] are mostly evaluated on datasets generated by several RL-trained policies of varying quality. In this study, we evaluate both offline IL and offline RL algorithms on datasets collected from one or more humans, which can both break the assumption of optimality in IL, and present interesting challenges for offline RL methods compared to commonly used RL-generated datasets.
Empirical Studies in Reinforcement and Imitation Learning: Prior work has benchmarked Reinforcement Learning (RL) algorithms in continuous control domains [65], run extensive evaluations of model-based RL algorithms [66] and on-policy RL methods [67], and shown how Deep RL algorithms can be extremely sensitive to hyperparameter choices and can make reproducing results challenging [68]. There are fewer empirical studies in imitation learning. The MAGICAL benchmark [23] consists of 2D environments that test the ability of IL algorithms to generalize. Both RLBench [24] and Ravens [25] provide several simulated robot manipulation tasks and expert demonstrations for imitation learning, but demonstrators are pre-programmed and rely on ground-truth simulator state. Simitate [69] is an imitation learning benchmark suite consisting of real-world motion trajectories collected by humans, and carefully translated into simulation using extensive sensor instrumentation in the real world. By contrast, our datasets are collected via remote teleoperation from humans in both simulated and real-world settings, allowing fast and easy demonstration collection for a wide range of tasks without assuming privileged instrumentation. Recently, Hussenot et al. [70] presented an empirical study on the importance of hyperparameter selection in imitation learningit consists of an extensive evaluation of both offline and online imitation learning algorithms using modest-sized datasets (~10s of trajectories). This work is complementary to ours -we also confirm the importance of hyperparameter selection when learning offline from human-provided datasets (~100s of trajectories).
Prior work has also established benchmarks for motion-based learning from demonstration methods [71,72], which directly model entire robot arm trajectories. Lemme et al. [72] evaluates motion generation methods by having a robot arm reproduce several point-to-point motions from demonstrations. This differs from our focus -tabletop manipulation tasks where a robot must interact with one or more objects. Lemme et al. [72] also evaluate the robustness of the learned motions by introducing perturbations -similar mechanisms could be used to understand the robustness of the policies trained in this study. While such evaluations are important for deploying policies in real-world settings, we leave this for future work. Similar to our study, recent work by Rana et al. [71] collected demonstration trajectories across many humans and tasks. However, their datasets consist of robot end effector trajectories, while our datasets and learning methods focus on leveraging additional modalities such as object poses and camera images to train policies that can solve tasks across several scene configurations. Rana et al. [71] also used crowdsourced humans to evaluate learned robot motions with subjective metrics such as safety, while we primarily evaluate our policies using task success rate. Subjective measures like safety are important for real-world policy deployment, but this is also left for future work.
Robot Manipulation Benchmarks: Several benchmark robot manipulation tasks have been proposed before [73-76, 24, 77]. Benchmark datasets [18,19] have also been proposed recently for batch (offline) reinforcement learning. While these datasets span a variety of domains (locomotion, control, autonomous driving, video games, robot manipulation), most of the datasets are generated by autonomous agents trained with online reinforcement learning. Unfortunately, this means that most of these datasets are limited to tasks that RL methods can solve from scratch. By contrast, we focus on datasets collected by one or more humans, allowing us to increase the complexity of the tasks we consider. Furthermore, our study explores how learning from human data can be substantially different than agent datasets.

B.1 Data Collection
Machine-Generated (MG) Datasets. We trained RL agents to solve tasks and we subsequently collected data from the trained agents to form our MG datasets. We only considered the Lift and Can tasks, as the other tasks were exceedingly difficult for RL to solve. Our RL algorithm is based on the Soft Actor-Critic [30] implementation in RLkit. We trained the algorithm with episode lengths of 150 and a batch size of 1024, and used dense rewards to facilitate exploration. We trained on the Lift and Can tasks for 2.4 million and 7.2 million environment steps respectively. For each task we saved agent checkpoints every 600k timesteps in training, which amounts to 5 checkpoints for Lift and 13 checkpoints for Can. Most checkpoints achieved 0% average task success rate but the last few checkpoints reached an average task success rate of ∼ 70 − 80%. For generating the datasets, we loaded each saved checkpoint and generated 300 rollouts of a fixed length of 150 timesteps. We annotated the transitions in the rollouts with sparse task completion reward and a "done" signal of True when the corresponding next state in the transition represented task success. Overall the Lift dataset consists of 225k transitions, and the Can dataset consists of 585k transitions.
Proficient-Human (PH) and Multi-Human (MH) Datasets. Datasets were collected by using the RoboTurk platform [15,17]. Our six human operators used smartphones to control robot arms hosted on a simulation server and were provided with video streams in a local web browser. The six operators were located at distances ranging from tens to several thousands of miles from the simulation server, and consisted of two "better" quality operators, two "okay" operators, and two "worse" operators.
Since the Transport task requires controlling two arms, we used Multi-Arm RoboTurk [36] to allow pairs of operators to collect data. The Transport (PH) dataset consists of 300 demonstrations collected jointly by the two proficient operators, while the Transport (MH) dataset consists of 6 sets of 50 demonstrations, where each set consists of a pairing of demonstrators (Better-Better, Okay-Okay, Worse-Worse, Worse-Better, Okay-Better, Worse-Okay). As explained in Sec 4.2, we also further split the MH datasets into smaller subsets to investigate how suboptimal human data affects performance.
Unlike the other tasks, all MH data subsets consist of exactly 50 demonstrations, corresponding to a specific pairing of demonstrators. Table 4 shows the average trajectory length by dataset, which is a proxy for the quality of the dataset -less experienced operators produced longer trajectories.
Can-Paired Dataset. A single experienced operator collected 2 demonstrations for each of 100 task initializations on the Can task, resulting in 200 total demonstrations. Each pair of demonstrations consists of a "good" trajectory, where the can is picked up and placed in the correct bin, and a "bad" trajectory, where the can is picked up, and tossed outside of the robot workspace. Since the task initializations are identical, and the first part of each trajectory leading up to the can grasp is similar, there is a strong expectation for algorithms that deal with suboptimal data, to be able to filter the good trajectories from the bad ones, and achieve near-perfect performance (for reference, BC-RNN can achieve near-perfect performance on the 50% subset of Can (PH), which corresponds to 100 good quality demos, see Fig 3).
Preparing Demonstration Subsets. In order to prepare smaller datasets used in Sec 4.6 and Fig 3, we sampled a fixed portion (20% and 50%) of the trajectories uniformly per human that provided data. This ensured that the smaller size datasets were not biased towards higher or lower quality data.
We also split all datasets and data subsets into training (90%) and validation (10%) subsets, using the same methodology. Models were not trained on the validation subsets.

B.2 Training Setup
As mentioned in Sec 3, each agent is trained for N epochs, where each epoch consists of M gradient steps, and evaluated every E epochs, by running 50 rollouts in the environment and reporting the success rate over a maximum horizon. All networks are trained using Adam optimizers [78]. The average trajectory length in each dataset (Table 4) was used to determine an appropriate evaluation rollout horizon for each dataset (Table 5). For each agent, we report the maximum success rate over the coarse of training, and average over 3 seeds. For agents trained with low-dimensional observations, N = 2000, M = 100, and E = 50, and for image observations, N = 600, M = 500, and E = 20. With the exception of the MG datasets, agents were only trained over 90% training subsets, with 10% held out as validation.

C Problem Setup and Algorithm Details
Consider a robot manipulation task, formulated as an infinite-horizon discrete-time Markov Decision Process (MDP), M = (S, A, T , R, γ, ρ 0 ), where S is the state space, A is the action space, T (·|s, a) is the state transition distribution, R(s, a, s ) is the reward function, γ ∈ [0, 1) is the discount factor, and ρ 0 (·) is the initial state distribution. At every step, an agent observes the state s t , uses a policy π to choose an action, a t = π(s t ), and observes the next state, s t+1 ∼ T (·|s t , a t ), and reward, r t = R(s t , a t , s t+1 ). The goal is to learn an policy π that maximizes the expected return: E[ ∞ t=0 γ t R(s t , a t , s t+1 )]. In this study, we assume access to an offline dataset of trajectories [12] and that a policy π θ must be learned offline, without collecting any additional samples from the MDP.
Rewards and Dones. Unless otherwise mentioned, the rewards used in this study are binary task completion rewards, R(s, a, s ) = 1[s ∈ G], where G ⊂ S is the set of all states where the task is considered to be solved. Similarly, the done signal, which indicates the end of an episode, is considered to be true for a transition (s, a, r, s ) if the task is solved in state s or if it is the last transition in a dataset trajectory. The reward and done signals are only used by Batch (Offline) RL algorithms (BCQ, CQL, IRIS).
We now outline the Imitation Learning and Batch (Offline) Reinforcement Learning algorithms used in our study.

C.1 Behavioral Cloning (BC, BC-RNN, HBC)
Behavioral Cloning [8] (BC) is a common method for learning a policy from a set of demonstrations. It trains a policy π θ (s) to clone the actions in the dataset via the objective: BC-RNN is a variant of BC that uses a Recurrent Neural Network (RNN) as the policy networkthis allows the policy to model temporal dependencies in the data through the recurrent hidden state. The network is trained on length-T temporal sequences of data (s t , a t , ..., s t+T , a t+T ). The network predicts the sequence of actions using the sequence of states as input. At test-time, the RNN policy network is unrolled one-step at a time, a t , h t+1 = π θ (s t , h t ) where h is the RNN hidden state. The hidden state is refreshed every T steps.
Hierarchical Behavioral Cloning (HBC) trains hierarchical policies and has been shown to be effective in learning from offline human demonstrations [20,10,36]. HBC consists of a low-level policy that is conditioned on future observations s g ∈ S (termed subgoals) and outputs action sequences to try and achieve them, and a high-level policy that predicts future subgoals from the current observation. The architecture and training procedure for the low-level policy π L θ (s, s g ) is nearly identical to BC-RNN -the only difference is the subgoal conditioning (during training, this is the final observation of the sampled sequence). The high-level policy π H θ (s) is trained to predict subgoal observations s t+T that occur T timesteps in the future from the current observation s t , and is often a conditional Variational Autoencoder (cVAE) [79] that learns a conditional distribution π H (s t+T |s t ). At test-time, the high-level policy is queried for a new subgoal every T timesteps, and the low-level policy is unrolled subsequently for T timesteps using the predicted subgoal.

C.2 Batch Constrained Q-Learning (BCQ)
Batch Constrained Q-Learning (BCQ) [26] is a commonly used algorithm for batch (offline) reinforcement learning [11,12]. It maintains a Q-network Q ψ (s, a), a generative action model p ω (a|s) (in the original implementation, this is a cVAE [79]), and (optionally) a perturbation actor network π θ (s, a). Fujimoto et al. [26] noted that target value estimation in batch RL can suffer from overestimation error due to querying the Q-network on actions unseen in the dataset. To address this, BCQ modifies the way target value estimates are constructed for the temporal difference Q-network loss by approximately constraining the Q-network maximization to actions seen in the dataset.
To form the target value estimate, actions are sampled using the generative model and perturbed using the perturbation actor A = {a i + π θ (s, a i )|a i ∼ p ω (·|s)} N i=1 , and then used to maximize the Q-network at the next state Q target = r + γ max a i ∈A Q ψ (s , a i ). The Q-network is trained by minimizing the temporal difference loss (Q ψ (s, a) − Q target ) 2 . As in DDPG [80], the perturbation actor is trained to maximize Q-values via the loss −Q ψ (s, a + π θ (s, a))|a ∼ p ω (·|s). Using the perturbation actor to modify the samples is optional -we find that removing it is often beneficial (see Appendix I), as in other prior work [42]. At test-time, N actions are sampled from the generative action model, perturbed by the actor (if present), and the action with the highest Q-value is selected.

C.3 Conservative Q-Learning (CQL)
Conservative Q-Learning (CQL) [27] is a recent batch (offline) RL algorithm that addresses the overestimation issue of Q-values directly. While other offline RL algorithms place constraints on the policy to stay within the support of data, CQL places an implicit constraint on the Q-function that lower-bounds its values. Specifically a Q-value regularizer is added to the policy evaluation objective to ensure that the estimated Q-values under the policy π θ (s, a) do not overestimate the Q-values under the data distribution µ(a|s): Kumar et al. [27] proved that this formulation lower bounds the true Q function and has theoretical improvement guarantees. CQL is simple to implement and has shown state-of-the-art empirical results on various offline datasets.

C.4 Implicit Reinforcement without Interaction at Scale (IRIS)
Implicit Reinforcement without Interaction at Scale (IRIS) [20] is a batch (offline) RL algorithm proposed for learning from large robot manipulation datasets collected by multiple humans. It is identical to Hierarchical Behavioral Cloning (HBC) except that the high-level policy consists of both a cVAE subgoal sampler and a value function trained using Batch Constrained Q-Learning (BCQ). Similar to BCQ, the high-level policy selects the subgoal by sampling N subgoals from the cVAE, and picking the state with the highest value estimate. Since the low-level policy is unimodal, all modeling of suboptimal and diverse data takes place in the high-level policy, at a reduced frequency (every T timesteps), enabling temporal abstraction.

D.1 Network Architecture
Here we briefly overview the general architecture design. Please refer to later sections for more detailed hyperparameter choices.
General Network Details. All Multi-Layer Perceptrons (MLPs) use ReLU activations. All Recurrent Neural Networks (RNNs) are 2-layer LSTMs, where the final layer hidden states are fed to downstream modules.
Encoding Observations. All networks have an observation encoder that processes observation dictionaries into a single vector. The encoder takes image observations, passes each through an observation-specific encoder into a low-dimensional vector, and finally concatenates the encoded image vectors with the low-dimensional observation vectors. For example, visuomotor policies typically contain two image encoders, one for the frontview camera, and one for the wrist camera. Each image encoder consists of a ResNet-18 network [46] followed by a spatial-softmax layer [45].
Observation-Conditioned Network Structure. Here we describe the general structure of networks that take observations as inputs. This includes policies, as well as value and state-action value functions, and comprises the majority of all networks used by the algorithms. First, observations are encoded (if they are images) and concatenated together into a flat vector. Next, in the case of networks that use an RNN (policies for BC-RNN, HBC, and IRIS), the flat encoded observations are sent through the RNN in order to produce hidden state outputs. Finally, outputs are passed to an MLP consisting of one or more layers in order to predict the item of interest. In the case of policy networks, the output either consists of raw actions, or the parameters of an action distribution (e.g. Gaussian Mixture Model parameters), while value functions output a single scalar. Policy action predictions (raw action predictions and mean parameter predictions) are passed through a tanh layer for normalization to [-1, 1].

D.2 Hyperparameter Selection Procedure
For each algorithm, we tuned hyperparameters separately for the Machine-Generated (MG) datasets, the Proficient-Human (PH) datasets, and the Multi-Human (MH) datasets. We used both the Lift (MG) and Can (MG) datasets for selecting a single set of hyperparameters for use on all MG datasets. We used the Square (PH) and Transport (PH) datasets for selecting a single set of hyperparameters for use on all PH datasets. We used the Square (MH) and Transport (MH) datasets for selecting a single set of hyperparameters for use on all MH datasets. We used Weights and Biases [81] to conduct hyperparameter tuning.
Note that the Tool Hang (sim) and all real tasks were purely used for evaluation purposes -no hyperparameter tuning took place on these tasks. This was to see whether our insights from hyperparameter tuning above could transfer to our hardest simulation task, and directly to real robot datasets.
Also note that we excluded HBC and IRIS from image-based training, due to the dependence of these algorithms on subgoal reconstructions, which could be problematic for high-dimensional images.
BC and BC-RNN. We scanned the following hyperparameters for BC and BC-RNN: • Learning rate: We compared 1e − 3 and 1e − 4 for both BC and BC-RNN. We found lower learning rate to perform better consistently.
• Actor network dimension: We scanned different dimensions for the action output network (MLP): [300, 400], [1000, 1000], and no MLP (directly output from RNN). We found higher capacity works better for BC, and no MLP works better for RNN. Our hypothesis is that RNN already has enough capacity to learn the tasks, and larger actor network results in overfitting.
• GMM for action output: Stochastic policy (GMM output) performs better than deterministic policy, although the gap is smaller for BC-RNN than for BC. We scanned different number of modes: {5, 10, 100} standard deviation minimum clipping: {1e − 2, 1e − 4, 1e − 6} and observed no significant difference in performance.
• Sequence length (RNN): We compared sequence length {10, 30, 50}. We found longer sequence length does not improve performance significantly. We opt for short sequence length for training efficiency. • RNN Hidden Dim: We compared different hidden dimension sizes for the RNN (LSTM): {100, 400, 1000, 2000}. We observed that 400 and 1000 tend to work well for most tasks.
BCQ. As in the original implementation [26], we used two critic networks. When using the perturbation actor, we also left the scale unchanged from the original (0.05). When using the VAE action sampler, we used a latent dimension of 14. We scanned the following hyperparameters: • Learning rate: For the learning rate of the critic (CLR) and the action sampler (ASLR), we compared 1e − 3 and 1e − 4. We found smaller learning rate for the action sampler usually has better performance on human datasets, while there was not much difference between the two learning rates for the critic.  [80]. As is shown in Table. 18, we found the performance drastically decreases when using the actor. By contrast, with the machine-generated data (MG), we found BCQ with actor enabled usually has better performance. • Action sampler (VAE vs GMM): We compared two types of action sampler -the VAE and GMM and found that the VAE generally has better performance than the GMM sampler. However, the GMM sampler also allows for a direct comparison with BC -see Table 19 for results.  N(0, 1), it can also be learned as part of the KL loss. We compared using the normal prior to using a state-dependent GMM prior, whose parameters are output by an MLP. We found that the normal prior N(0, 1) generally performs better than the learned GMM prior, except for the Multi-Human(MH) dataset. Therefore, we opted the learned GMM prior only for Multi-Human(MH) dataset and keep N(0, 1) prior for the others. • Tau (target network update rate): We compared 5e − 3 and 5e − 4 and found the lower value of 5e − 4 to give better results.  [27], we experimented with a deterministic Bellman backup objective -i.e. removing the additional entropy term from the target Q value backup. We found that the deterministic backup outperformed the non-deterministic backup, though at times the gains were marginal. We subsequently chose to deterministic variant.
• BC start steps: We experimented with replacing the policy loss objective for CQL with the behavior cloning objective for the first 40000 gradient steps of training. We found that the performance of the agent degraded after these initial gradient steps, so we set this hyperparameter to 0 in all subsequent experiments.
• Batch size: For low-dim experiments, we found that increasing the batch size from the default value of 100 to 1024 can lead to significant gains in stability and performance. For image experiments we were bottlenecked by GPU memory and we used a batch size of 8.
• Lagrange variant: We found that the Lagrange variant consistently performed better than the non-Lagrange variant and was less sensitive to hyperparameters. We therefore decided to use the Lagrange variant in all of our experiments.
• Lagrange threshold τ: We found that a threshold of τ = 1 often caused the dual weight to diverge to large values. We subsequently experimented with higher values of 5, 10, 25 and found that the algorithm was relatively stable with all of these with no major difference in performance. We subsequently chose τ = 5.
HBC. We scanned the following hyperparameters for HBC: • Learning Rate: We scanned 1e − 3, 1e − 4 for both the policy and goal learning rate, and generally found the higher learning rates to perform better.
• VAE vs AE: We compared using a VAE vs. only an AE for the planner network, and found the best performing VAE model outperformed the best performing AE model.
• VAE Prior: We compared using N (0, 1), Learned GMM priors, and found that, when tuned, the GMM prior worked the best.
• VAE Latent Dim: We compared 2, 16, 100, and found that 16 worked the best. IRIS. We first initialize our scan with the best hyperparameters from HBC and BCQ. Note that from our BCQ scan, we use Action Sampler Learning Rate = 1e − 4 and Num Action Samples (Train/Test) = 10/100. We also use Tau = 5e − 4 and no value actor for PH and MH datasets and Tau = 5e − 3 with value actor enabled for MG datasets. We then proceeded to scan over the following hyperparameters: • Learning Rate: We compared 1e − 3, 1e − 4 for the policy, goal, and value learning rates individually, and generally found the higher learning rates to perform better. The exception is the multi-human setting, where we found a lower Value LR to work better.
• Value KL Weight: We compared 0.5, 0.05, and found that both values generally performed similarly.
• Value Actor: We compared using a value actor True, False on the MG datasets, and found that an actor can improve results on the Can MG dataset.

D.3 Final Hyperparameters
We present our finalized set of hyperparameters in Tables 6 and 7 for BC, Tables 8 and 9 for BC-RNN,  Tables 10 and 11 for BCQ, Table 12 for HBC, Table 13 for IRIS, and Tables 14 and 15 for CQL. Each column shows the hyperparameters for learning from a dataset setting (shared across environments): PH for proficient human, MH for multiple humans, MG for machine-generated. -means the hyperparameter is inherited from the default hyperparameters shown on the left-most column. For further details on the training setup shared by all algorithms (such as the number of gradient steps and epochs), see Appendix B.2.

D.4 Additional Details on Hyperparameter Choice Study for BC-RNN
In Sec 4.4, Fig 2b, and Fig 2c, we presented results that showcase how changing a subset of BC-RNN hyperparameters can result in large performance decreases. In this section, we provide more details for each change.
Larger LR. We changed the policy learning rate from the default 1e-4 to 1e-3.
By default, all BC-RNN policies on PH and MH learned a Gaussian Mixture Model (GMM) distribution. Here, we replace the GMM distribution with a direct action prediction.
Larger MLP. As noted in Appendix D.1 above, there is an MLP that transforms RNN hidden states into action (or action distribution) predictions. By default, we use a single linear layer, but here we try adding two layers of size 1024.
Shallow Conv. As noted in Appendix D.1 above, all image encoders are ResNet-18 networks. Here, we tried replacing the ResNet with the shallow convolutional network from Finn et al. [45].
Smaller RNN Dim. By default, we use a hidden layer size of 400 for low-dimensional datasets, and 1000 for image datasets. Here, we tried reducing the dimension to 100 for low-dim, and 400 for image.

E.1 Simulation Tasks
All simulation tasks were designed using MuJoCo [82] and the robosuite framework [76]. We used Panda robotic arms in both simulation and the real world for this study. The action space for the agent is a 7-dimensional vector for each arm where the first 3 coordinates are the desired translation from the current end effector position, the next 3 coordinates encoder the desired delta rotation from the current end effector rotation, and the final coordinate controls the opening and closing of the gripper fingers. The delta rotation is encoded in axis-angle form, where the norm of the 3-vector is the angle, and normalizing the 3-vector produces the axis. The policy outputs actions at a rate of 20 Hz. Policy actions are transformed into end effector target poses and sent to an operational space controller [83] that outputs the robot joint torques at 500 Hz to try and achieve the desired cartesian poses.
All simulation environments will be released along with datasets and codebase upon publication. We next describe the low-dimensional object observations and initial state randomization for each task.
Lift. Object observations (10-dim) consist of the absolute cube position and cube quaternion (7-dim), and the cube position relative to the robot end effector (3-dim). The cube pose is randomized at the start of each episode with a random z-rotation in a small square region at the center of the table.
Can. Object observations (14-dim) consist of the absolute can position and quaternion (7-dim), and the can position and quaternion relative to the robot end effector (7-dim). The can pose is randomized at the start of each episode with a random z-rotation anywhere inside the left bin.
Square. Object observations (14-dim) consist of the absolute square nut position and quaternion (7-dim), and the square nut position and quaternion relative to the robot end effector (7-dim). The square nut pose is randomized at the start of each episode with a random z-rotation in a square region on the table.
Transport. Fig E.1 (top) shows a full demonstration of the task. Object observations (41-dim) consist of the absolute position and quaternion of the hammer (7-dim), the absolute position and quaternion of the trash cube (7-dim), the absolute position and quaternion of the lid handle (7-dim), the target bin position (3-dim), the trash bin position (3-dim), the relative positions of the hammer and the lid handle with respect to the first arm end effector (6-dim), the relative positions of the hammer and trash cube with respect to the second arm end effector (6-dim), a binary indicator for the hammer reaching the target bin (1-dim), and a binary indicator for the trash reaching the trash bin (1-dim).
The position of all bins, the lid, the trash cube, and the hammer are randomized in small squares at the start of each episode. The z-rotation of the trash cube and the hammer are also randomized with a full range of 108 and 60 degrees respectively.
Tool Hang. Fig E.1 (middle) shows a full demonstration of the task. Object observations (44-dim) consist of the absolute position and quaternion and relative pose and quaternion with respect to the end effector of the base frame (14-dim), the insertion hook (14-dim), and the ratcheting wrench (14-dim), as well as binary indicators for whether the stand was assembled (1-dim) and whether the tool was successfully placed on the stand (1-dim). The position of the insertion hook and ratcheting wrench and z-rotation (range of 40 degrees) are randomized in a small square at the beginning of the episode.

E.2 Real World Task Setup
We first describe details about the robot workspace and setup. Next, we discuss the materials needed to construct each physical task. We took care to approximately match the visual appearance, the physical dimensions, and the task initialization randomizations of the real tasks to those in simulation.
Workspace and Setup. Our physical robot workspace consists of a Franka Emika Panda robotic arm, a front-view Intel RealSense SR300 camera, and a wrist-mounted Intel RealSense D415 camera. The robot arm and the front-view camera are attached rigidly to the table, while the wrist-view camera points towards the space in between the robot gripper fingers. Demonstration data was collected from the robot sensors and the two cameras at approximately 20 Hz. Similar to simulation, the robot is controlled using an operational space controller, using the same action space as the one in simulation (see above).
Lift (Real). A white 3D-printed cube that measures 4 cm in all dimensions was used for this task, with a small initialization square that roughly corresponds to the one for the simulation task.
Can (Real). We purchased this tray for the left bin (where the can starts in each episode), and four of these small boxes to construct the right bin. We used a 7.5 oz Coca Cola Zero Sugar Diet Soda Can (empty, and stuffed with some paper) as the coke can.
Tool Hang (Real). Fig E.1 (bottom) shows a full demonstration of the task on the real robot. We purchased this handbag stand, and sawed the base rod and the hook rod in half. We also outfitted the bottom of the hook rod with a soldering iron tip using 2-part epoxy in order to make the hook more amenable to insertion. We also purchased this ratcheting wrench tool set and used the 17mm-19mm wrench for the task.  In this section, we provide additional details related to the observation spaces used to train agents (shown in Fig E.2). Both low-dim agents and image agents receive proprioception observations (9-dim per arm) consisting of the end effector position (3-dim), quaternion (4-dim), and gripper finger positions (2-dim). The low-dim agents also receive object observations (described above, see Appendix E.1), while image agents receive an external camera image and a wrist camera image per robot arm. We first provide further details on cameras and image sizes used per task, then we discuss pixel shift randomization details (shown to be crucial for visuomotor learning, see Sec 4.3 and Fig 2a), and finally, we provide more details on the experiments in Sec 4.3.

E.3 Observation Space Details
Image Observations per Task. For all tasks except Transport, we provide two image observations -one from a front-view camera and one from a wrist-mounted camera. For the Transport task, we provide four image observations to the agent -two from shoulder-view cameras per arm, and two from wrist-mounted camera on each arm. The front-view and shoulder-view cameras are the same cameras used by human operators to provide task demonstrations. All simulation tasks, with the exception of Tool Hang, provide 84 by 84 images. All real robot tasks, with the exception of Tool Hang, provide 120 by 120 images. Tool Hang, in both simulation and real world provides 240 by 240 images (due to the need for high-precision control). On the real robot, raw camera frames are read from the camera at a full resolution of 640 by 480, then a center crop of 480 x 480 is applied, and finally, images are resized to the appropriate resolution.
Additional Details for Pixel Shift Randomization. To implement random pixel shifts [84][85][86][87] for input image observations, we take large random crops from the source images when feeding image observations to any network. Additional Details for Observation Space Study. Here, we describe the observations added for the experiments presented in Sec 4.3 and Fig 2a. When adding EEF Vel observations, we added linear end effector velocity (3-dim per arm), angular end effector velocity (3-dim per arm), and gripper finger velocities (2-dim per arm). When adding Joint observations, we encoded the joint positions using cosine (7-dim per arm) and sine (7-dim per arm), and also provided joint velocities (7-dim per arm).

G Additional Results on Policy Selection
In this section, we present some more results and discussion on offline policy selection related to Sec 4.5 and Fig 4a. We first show that policy performance can vary substantially during a training run -this is why policy selection is non-trivial. One might expect that adding more validation data could help improve its use as a selection criteria for selecting a good policy. We empirically show that this is not necessarily true. Finally, we show that success rate can keep climbing, even while the validation loss increases substantially.
Policy checkpoints can vary substantially in performance during training, even when performance appears to converge. Fig F.1 shows several different learning curves for BC-RNN agents on our human datasets. Low-dimensional agents exhibit significant variance in policy success rate even in later stages of training, on harder tasks like Square (see Fig F.1c and Fig F.1g) and Transport (see Fig F.1d and Fig F.1h). While this is also true for image agents (see Fig F.1k and Fig F.1l), even simpler tasks like Lift (Fig F.1i) and Can (Fig F.1j) can suffer from such variance in performance. This kind of variance in performance is problematic for real world settings where it's not feasible to run 50 rollouts per checkpoint for each training run, as we have done in simulation. This makes offline policy selection a difficult, but important problem to solve.
Increasing the amount of validation data does not improve policy selection using validation loss. Fig 4a used a validation dataset size that was 10% of the collected data. We also tried training low-dim BC-RNN policies on the Square (PH) and Transport (PH) datasets, where we used 30% of the collected data for validation (and only 70% for training). Across 3 seeds, the best policy on Square (PH) achieves 80.7 ± 0.9, while the policy achieving the lowest validation loss achieves 2.7 ± 1.9, and the best policy on Transport (PH) achieves 64.0 ± 2.8 while the policy achieving the lowest validation loss achieves 0.7 ± 0.9.
Success rate can increase even while validation loss increases substantially. Empirically, we found that best validation loss occurs relatively early in training (epoch 100-300) but the best performance occurs much later. In Fig G.1, we present selected plots of success rate and validation loss versus epoch, to show this. The plots also show that validation loss can keep increasing substantially in later epochs -despite this, the success rate also keeps increasing. This further shows that validation loss is a poor measure of policy performance.  . We present success rates averaged over 3 seeds for each method across the image Machine-Generated (MG) datasets. BCQ outperforms the other methods; however, it is possible that recent batch RL methods [38,43] that have been shown to work on pixel observations might be able to perform even better. Table 16 shows results on the Machine-Generated (MG) datasets with image observations. BCQ outperforms the other methods; however, it is possible that recent batch RL methods [38,43] that have been shown to work on pixel observations might be able to perform even better.  Table 17: Results on Suboptimal Human Data (Image). We present success rates averaged over 3 seeds for each method across different subsets of the Multi-Human datasets, corresponding to mixtures of demonstrations from "Better", "Adequate", and "Worse" human operators, and finally on a diagnostic dataset with paired success and failure human trajectories for each starting initialization. We omitted Batch RL methods (except for Can-Paired) due to their poor performance on the other human datasets with image observations. Table 17 shows results on our multi-human data subsets with image observations. We excluded batch RL methods due to their poor performance on human datasets with image observations. BC-RNN improves over BC on all datasets, especially datasets with lower quality data.  4.0 ± 0.0 Table 19: BCQ Hyperparameter Sensitivity -matching parameters to BC. We find that matching the hyperparameters of the BCQ action sampler to the ones we used for BC is not sufficient to improve performance.

H.2 Suboptimal Human Datasets
In Table 18, we show that using the BCQ perturbation actor (see Appendix C.2) can have a catastrophic effect when training on human datasets. The results show that there is a large performance drop after enabling the perturbation actor (over 80% on Can-PH, for example). This result showcases BCQ is highly sensitive to the perturbation actor.
In Sec 4.1 and Sec 4.2, we empirically saw BCQ consistently underperform compared to BC. To further investigate this problem, we tried matching the hyperparameters of the BCQ action sampler to the BC model we used (the same learning rate, MLP architecture, and using a Gaussian Mixture Model). We present the results in Table 19. BCQ still underperforms -since the only difference between BC and this version of BCQ at test-time is selecting a GMM action sample uniformly at random versus using the Q-function to select one, this indicates that the Q-function is responsible for poor performance.  Table 20: CQL Hyperparameter Sensitivity. We find that CQL (1) is highly sensitive to learning, (2) benefits greatly from larger batch sizes, and (3) benefits from the deterministic backup and Lagrange variants with significant improvements on some datasets.
We investigate the effect of various hyperparameters on CQL in Table 20 for the low-dim Lift MG and PH datasets. First, we see that a smaller learning rate (specifically 10× smaller) for the Q and policy networks leads to a decrease in success rate of over 50%, indicating that CQL is highly sensitive to learning rate. Next, we see that a smaller batch size (specifically 100 instead of 1024) leads to a performance decrease of over 30%, indicating the CQL can greatly benefit from higher batch sizes. The results for excluding the Lagrange variant and the deterministic backup also indicate that these components can help, with substantial improvements in performance for the MG dataset yet marginal improvements for the PH dataset.  As in Acme [88], when using Gaussian Mixture Model (GMM) policies during rollouts, we ignore the learned standard deviations of each mode and instead set it to 1e-4. This amounts to sampling one of the GMM modes instead of sampling from the full GMM distribution. A similar trick is often used when learning Gaussian policies, where at test-time, the mean action of the distribution is used instead of sampling from the distribution. In the table above, we present an ablation for not using this "low-noise-evaluation" (LNE) trick. In most cases, the performance decreases slightly. In early experiments (where the minimum learned std for each Gaussian mode was set to 1e-2 instead of 1e-4), this trick made a more substantial difference. We suggest using this trick by default in all experiments involving GMM policies.

K Additional Results on Multi-Human Datasets
This section contains additional results on Multi-Human datasets that were excluded from the main text for space reasons. This includes conducting the Observation Space study on Multi-Human datasets (the main text included results on the Single-Human datasets in Fig. 2a and Table 25), and results on the Lift and Transport Multi-Human data subsets (the main text only included results on the low-dim Can and Square subsets in Table 2, and the image subsets in Appendix H).

L Full Tables
This section contains more detailed tables that were excluded in the main text. These tables correspond to the results in Fig 2, Fig. 3 Table 29: Effect of Policy Selection Criteria. We compare how performance decreases when choosing the policy to evaluate by using the lowest validation loss, or when using the final trained checkpoint, compared to the best performing policy. Corresponds to results in Fig 4a.