{'fraction_non_alphanumeric': 0.03161421911421911, 'fraction_numerical': 0.007925407925407926, 'mean_word_length': 4.884945130315501, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation

Learning complex manipulation tasks in realistic, obstructed environments is a challenging problem due to hard exploration in the presence of obstacles and high-dimensional visual observations. Prior work tackles the exploration problem by integrating motion planning and reinforcement learning. However, the motion planner augmented policy requires access to state information, which is often not available in the real-world settings. To this end, we propose to distill a state-based motion planner augmented policy to a visual control policy via (1) visual behavioral cloning to remove the motion planner dependency along with its jittery motion, and (2) vision-based reinforcement learning with the guidance of the smoothed trajectories from the behavioral cloning agent. We evaluate our method on three manipulation tasks in obstructed environments and compare it against various reinforcement learning and imitation learning baselines. The results demonstrate that our framework is highly sample-efficient and outperforms the state-of-the-art algorithms. Moreover, coupled with domain randomization, our policy is capable of zero-shot transfer to unseen environment settings with distractors. Code and videos are available at https://clvrai.com/mopa-pd


Introduction
Solving complex manipulation tasks in obstructed environments is a challenging problem in deep reinforcement learning (RL) since it requires precise object interactions as well as collision-free movement across obstacles. To tackle this problem, prior works [1][2][3] have proposed to combine the strengths of motion planning (MP) and RL -safe collision-free maneuvers of MP and sophisticated contact-rich interactions of RL, demonstrating promising results. However, MP requires access to the geometric state of an environment for collision checking, which is often not available in the real world, and is also computationally expensive for a real-time control. To deploy such agents in realistic settings, we need to resolve the dependency on the state information and costly computation of MP, such that the agent can perform a task in the visual domain.
To this end, we propose a two-step distillation framework, motion planner augmented policy distillation (MoPA-PD), that transfers the state-based motion planner augmented RL policy (MoPA-RL [1]) into a visual control policy, thereby removing the motion planner and the dependency on the state information. Concretely, our framework consists of two stages: (1) visual behavioral cloning (BC [4]) with trajectories collected using the MoPA-RL policy and (2) vision-based RL training with the guidance of smoothed trajectories from the BC policy. The first step, visual BC, removes the dependency on the motion planner and the resulting visual BC policy generates smoother behaviors compared to the motion planner's jittery behaviors. Then, in the second step, we further improve the visual policy Figure 1: Our two-step framework distills the state-based MoPA-RL agent into a visual control policy. In stage 1, we learn a visual BC policy using MoPA-RL trajectories, thereby ensuring smooth trajectories for the expert replay buffer R e , while removing the motion planner. In stage 2, we learn a BC trajectory-guided vision-based RL agent with an asymmetric actor-critic algorithm using both the expert replay buffer R e and agent replay buffer R π . To improve sample efficiency, we initialize the actor and the critic of the vision-based agent with the weights of the BC policy and the MoPA-RL critic, respectively.
using asymmetric actor-critic RL [5] directly from the image observations to overcome sub-optimality in the MoPA-RL policy through self-exploration.
For efficient vision-based RL training, our method leverages the experience of the BC policy and the state-based critic from MoPA-RL by initializing the critic network with the MoPA-RL critic and the actor network through BC pre-training; BC-trajectory guided RL training using a separate expert replay buffer; tuning the entropy coefficient to encourage exploitation (i.e., maximizing reward) over exploration (i.e., maximizing entropy).
In summary, our contributions are as follows: • We propose a novel framework to learn a visual control policy in cluttered scenarios by distilling a state-space policy that uses a motion planner into an image-space policy by removing the motion planner dependency.
• Our asymmetric visual policy leaning ensures high sample-efficiency using weight initialization followed by BC trajectory-guided RL with entropy coefficient tuning.
• The distilled visual policy is capable of zero-shot domain transfer to unseen environments with visual domain randomization during training.
• Our method outperforms the state-of-the-art methods in terms of success rate, sample efficiency, and path length on three manipulation tasks in obstructed environments.

Related Work
The objective of our work is to learn complex manipulation tasks in realistic and obstructed environments. Motion planner augmented RL methods [1][2][3] have shown promising results in solving complex tasks in obstructed environments by combining MP and RL. However, these methods cannot be deployed in real-world settings due to their dependency on state information. To remove this dependency, state-based policies must be transferred to the visual domain.
A typical approach to distill state-based policies into vision-based policies is behavioral cloning (BC [4]). However, BC often suffers when encountering states unseen during training [6]. Therefore, prior works have proposed various methods to improve BC agents using offline RL [7] and online RL [8]. However, these works do not involve the distillation of motion planners.
Several recent efforts have been made towards distilling MP algorithms into neural motion planners using learning-based methods [9][10][11][12][13]. Most of these efforts can be broadly categorized into imitation learning (IL) and RL paradigms. For IL, supervised learning on MP trajectories has been used to learn neural network-based planners [10][11][12][13][14]. However, the performance of such supervised learning approaches are limited by the collected dataset and demonstrator's performance.
To discover better policies with additional interactions, various off-policy RL methods have been studied for superseding motion planners with neural network policies [13,15,16]. These approaches utilize expert trajectories stored in the replay buffer for guided exploration, which leads to better policies than experts [17][18][19][20]. However, these works learn either in unobstructed environments [17][18][19][20] or on simpler tasks not involving object manipulation [13]. They also assume fully observable environments and learn policies in state space. In this work, we focus on complex manipulation tasks in obstructed environments using visual observations.
Moreover, most offline RL approaches [21,22] in sparse reward scenarios are coupled with hindsight experience replay (HER [23]). They empirically show that not using HER significantly hurts the agent's performance [13,5,16,17]. However, HER is designed to efficiently train goal-conditioned policies for multi-goal RL, which makes it unsuitable for tasks with multiple sequential steps without explicitly conditioning on goals. In contrast, our method successfully learns composite manipulation tasks in cluttered environments without using HER for tackling sparse rewards.

Method
Our goal is to solve complex manipulation tasks in obstructed environments with visual inputs. While MP-based methods can efficiently solve such tasks, they are restricted to learning a state-based policy which is difficult to transfer to real environments. Moreover, sampling-based motion planning incurs significant computational costs, making it hard to integrate them into real-time controllers. Thus, we propose a method that distills an MP-augmented state-based agent into a visual policy, removing the MP and state dependency. We formally define our problem and introduce the MP-augmented RL in Section 3.1. Then, we describe our two-step visual distillation approach in Section 3.2.

Preliminaries
Problem formulation We formulate our problem as a Markov Decision Process (MDP) M defined by a tuple (S, O, A, P, R, ρ 0 , γ) that consists of a state space S, partial observation space O (visual inputs corresponding to states), action space A, transition function P : S × A × S → R, reward function R, initial state distribution ρ 0 , and discount factor γ. The agent is represented as a policy π(a t |o t ), which takes an action a t under a visual observation o t and receives a reward r t = R(s t , a t ), with the state transitioning to s t+1 . The goal of the agent is to maximize the expected discounted sum of rewards E (s,a)∼π where T is the episode horizon.
Motion Planner-Augmented RL (MoPA-RL) To tackle manipulation tasks in cluttered environments, Yamada et al. [1] proposed to incorporate MP and RL via an MP-augmented MDP M = (S,Ã,P ,R, ρ 0 , γ) that consists of the state space S, enlarged action spaceÃ augmenting the direct action space A with the MP action spaceÂ, 5 augmented transition functionP : S ×Ã×S → R, augmented reward functionR(s,ã), initial state distribution ρ 0 , and discount factor γ.
MoPA-RL [1] learns a policy π φ (ã t |s t ) on the augmented MDPM with an off-policy RL algorithm, SAC [24]. Precisely, given a state s t , the policy predicts an actionã t , which is defined as a robot joint displacement ∆q t . If the action lies within the direct action space A, it is directly executed by the controller as the agent performs sophisticated and contact-rich manipulations. However, when the actions are larger in magnitude (i.e.ã ∈Â), the probability of collisions in the presence of obstacles increases. Therefore, a sampling-based motion planner, RRT-Connect [25], is called to realize such actions with large joint displacements by computing collision-free paths. This allows the agent to efficiently explore obstructed environments while avoiding collision [1].

Motion Planner Augmented Policy Distillation
Despite the advantages of MoPA-RL, its applicability to the real world is limited due to the high computational cost of MP and dependency on fully observable environment states. To this end, we propose a visual distillation method, motion planner augmented policy distillation (MoPA-PD), that learns an image-based control policy (without MP) from a state-based MP-augmented policy by leveraging its rollouts and the learned state-based critic as a guidance. Concretely, our proposed method consists of two stages, as illustrated in Figure 1. Given a state-based MoPA-RL agent, we first use BC to train a vision-based actor with trajectories collected from the MoPA-RL policy (Section 3.2.1), and then we further improve the vision-based agent via BC trajectory-guided asymmetric actor-critic RL [5] with the visual BC actor and MoPA-RL critic (Section 3.2.2).

Stage 1: Visual Behavioral Cloning and Trajectory Smoothing
The MoPA-RL policy is learned with access to complete information about the environment states and also utilizes MP for executing large action steps without collisions. In this paper, we aim to utilize demonstrations from this planner-based policy and distill it into a visual control policy, thereby deducting expensive MP computations and training the actor in visual domain. With the learned MoPA-RL policy, we first collect multiple transitions d i in low-level (direct) action space and store them into the MoPA-RL demonstration dataset Then, we train a visual BC actor π θ (a t |o t ) using observation-action pairs (o i , a i ) from the dataset D mp by minimizing the mean squared error.
Distilling the MoPA-RL trajectories using BC not only enables the BC actor to work directly on visual inputs but also reduces jerky motion planning behaviors of MoPA-RL, which occur due to motion planner's priority on obstacle avoidance over trajectory smoothness [26]. By removing unnecessary, jerky movements through visual BC, the resulting trajectories become smoother and even shorter, and thus help learning consistent and smooth motions.
However, the BC actor often fails when it encounters states not seen during training due to covariate shift [6]. To improve the robustness of the policy, we further train the visual BC actor using RL with additional environment interactions. In other words, the BC actor can be a good starting policy for visual RL training and the trajectories collected from the BC actor can effectively guide exploration.

Stage 2: Vision-based RL with Asymmetric Actor-Critic
In the second stage, we further train the visual policy using RL directly on image observations to enhance the robustness of the policy and overcome sub-optimality in the planner-based policy. For efficient RL training, we adopt an asymmetric actor-critic architecture [5], where the actor acts based on environment images and robot joint states while the critic learns from environment states. This architecture is motivated by learning transferable visual policies, which do not require state information during inference but benefit from them while learning in simulation. Our training procedure comprises of the following components: Weight initialization of actor and critic networks Initializing actor and critic networks with suitable weights can accelerate RL training by providing more optimal rollouts from the actor and informative learning signals from the critic compared to the randomly initialized actor and critic [27,28]. Especially when learning from pixels, suitable initialization can significantly improve sample efficiency [20,29]. To this end, we propose to leverage the state-based agent's experience by utilizing its weights for initializing our asymmetric visual agent's networks, thereby bringing their initial distributions closer. Thus, the visual actor network π θ (a t |o t ) is initialized with the weights of the visual BC actor learned in Section 3.2.1. We also initialize the critic Q ψ (s t , a t ) with the critic of MoPA-RL. Note that the action spaces of two critic networks are different, but initialization is still feasible since the action space of MoPA-RL is a superset of the direct action space.
BC trajectory-guided asymmetric RL After initializing the actor and critic networks, we collect agent rollouts into the agent replay buffer R π . As an RL algorithm, we adopt Soft Actor-Critic (SAC [24]), an off-policy model-free RL algorithm for continuous control. We optimize our asym- metric visual policy π θ by maximizing the following objective: where the temperature parameter α balances between exploration and exploitation using entropy H.
RL training with a BC-initialized policy often quickly deviates from the original policy. To prevent this problem and guide exploration during RL, we update our RL agent not only with agent trajectories but also with BC policy (i.e. expert) trajectories [18]. Thus, we collect smoothed trajectories from our visual BC policy, in an expert replay buffer R e . Then, for each RL training iteration, we sample 1:3 transitions from R e and R π , respectively. A separate expert replay buffer ensures guided exploration from the expert and also circumvents catastrophic forgetting after weight initialization. Note that we use the BC trajectories R e instead of the MoPA-RL data D mp because jittery motion planning paths in the MoPA-RL data make RL training difficult and sub-optimal.
Entropy coefficient tuning The performance of SAC is known to be sensitive to α [30]. Since our asymmetric agent already starts with the well-trained BC actor and MoPA-RL critic, we initialize α to values lower than the final α obtained in MoPA-RL. This is to ensure that with prior knowledge in the state-based agent, the visual agent focuses more on maximizing rewards over exploration. We describe the hyperparameter choice and implementation details in Section A.1 and Section B.
In summary, we propose a two-step visual distillation method for a state-based MoPA-RL agent using visual BC followed by BC trajectory-guided vision-based RL to remove the dependency on MP and environment states. For efficient RL training, our method leverages weight initialization of the actor and critic, the expert replay buffer of BC smoothed trajectories, and entropy coefficient tuning.

Experiments
In this paper, we propose to distill a motion planner augmented policy into a visual control policy for complex manipulation tasks in obstructed environments. In our experiments, we aim to answer the following questions: (1) Does IL efficiently learn policies for obstructed environments in image space? Moreover, is naively combining IL with RL sufficient for solving complex tasks? (2) Is distillation better than directly learning a visual policy using MP? (3) Does our approach, MoPA-PD, efficiently learn a visual policy using prior state-based exploration knowledge? (4) Is our visual policy capable of domain transfer and robust to unseen distractors?

Environments
We evaluate our approach on three obstructed environments (see Figure 2) from Yamada et al. [1], simulated using the MuJoCo physics engine [31]. We use a 32x32 image as a visual observation.   Table 1: Average success rate (ASR), episode length (AEL), and discounted return (ADR) of our method and baselines averaged over five seeds. Each method is evaluated after 3M environment steps. MoPA-RL [1] is the expert agent trained with state-based policy. All baselines below horizontal line are trained with asymmetric actor-critic for fair comparison. Maximum horizon is 250 for all tasks.
• Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside the box, needs to reach an object placed inside the box and push it to the goal location within the box. • Sawyer Lift: The Sawyer arm must reach an object placed inside a box, grasp, and lift it outside the box, avoiding collisions. The Sawyer arm is always initialized outside the box. • Sawyer Assembly: The Sawyer arm needs to assemble the fourth leg (already attached to the gripper) of a table into the vacant hole while avoiding collisions to other three legs. This environment is built upon the IKEA furniture assembly environment [32].
For Sawyer Push and Sawyer Assembly, the agent receives a reward proportional to the distance between the end-effector and object, or the object and goal position when the distance is less than . We use = 0.1 for Sawyer Push and = 0.3 for Sawyer Assembly while the distance between initial and goal state is around 1.
2. An episode is considered successful when the distance between the cube and goal (Sawyer Push) or the peg-head and goal (Sawyer Assembly) is less than 0.05. For Sawyer Lift, we use the reward function similar to Fan et al. [33], where a reward is defined for all three intermediate stages of the task: reach, grasp and lift. A bonus reward signal of 150 is received upon successful task completion in all the environments. We refer readers to Section D for more details.

Baselines
We compare our method with the following baselines to evaluate the merits of our approach: •

Evaluation
We compare our approach against baselines on the following evaluation metrics averaged over five random seeds and 100 unseen episodes per seed: • Average Success Rate (ASR) is the average number of successful episodes.
• Average Episode Length (AEL) is the average length of successful episodes.
• Average Discounted Return (ADR) is the average discounted sum of rewards T −1 t=0 γ t R(s t , a t ), with T being the episode horizon. An episode completed in more time steps has a lower discounted reward, due to exponentially discounted reward with γ = 0.99. Figure 3 illustrates the learning curves of our method and all other baselines using discounted rewards against the number of training steps. As per the trend, our method is far more sample-efficient and outperforms all the baselines. Asym. SAC [5] fails to directly learn a visual policy for Sawyer Lift and Sawyer Assembly in obstructed environments, where exploration is hard. Moreover, MoPA Asym. SAC, which is a direct attempt at learning a visual policy while using MP, does not successfully learn to solve the tasks. CoL [29] which has a straightforward combination of RL and BC objective for actor optimization receives partial rewards for some tasks at around 1.5-2M steps, much slower than our method's convergence.

Comparisons with baselines
As reported in Table 1, BC achieves decent performance for Sawyer Push and Assembly, but not for the Sawyer Lift task. However, it achieves high AEL and low ADR for all tasks, showing its inefficiency to solve the task fast. This is because BC is bound to perform as good as the expert demonstrations (here from MoPA-RL) at its best. In contrast, our method optimizes trajectories beyond expert signals and achieves significantly lower AEL and higher ADR compared to baselines.
Ablation on BC trajectory smoothing Our method without BC smoothing has higher variance across seeds and converges slower than our method for two tasks (see Figure 3). For Sawyer Assembly, it does not learn to solve the task at all. In short, BC smoothing of the motion planner trajectories is important for our method to work across all environments. This is because the motion planner based trajectories are usually jittery and non-smooth. Using behavioral cloning refines each transition, thereby making the entire trajectories much smoother and help in efficiently training the RL agent.
Ablation on weight initialization We compare our method with and without actor-critic initialization in appendix, Figure 7 and do not observe any episode success until 1.2M environment steps for the latter. This shows the importance of our proposed weight initialization for learning in the visual domain. We further elaborate the ablation results in Section A.2.
Ablation on entropy coefficient tuning We examine the effect of different values of α on the trade-off between entropy maximization and reward maximization for the SAC objective as shown in Section A.1. Compared to higher α values, smaller values of α improve the sample efficiency during visual policy learning. Since we utilize prior knowledge from the state-based agent, we use a smaller alpha to exploit the previously acquired knowledge instead of exploring the entire state space again.    Figure 4.

Policy Transfer to Different Domains
In this experiment, we learn our policy with domain randomization (DR) to verify its domain transfer capabilities. DR is a promising approach for modelling transferable policies using a multitude of variations in simulation during training [34,35], which makes the policy invariant to changes that are insignificant for the task. For this work, we randomize the simulation environment using different textures, colors, and lighting conditions for training as shown in Figure 4 and Figure 10. We then test our DR policy on three unseen scenarios, as illustrated in Figure 4 comprising of (1) the original environment without any randomization; (2) Scenario 1 with realistic background distractions (e.g., furniture, walls, and floor) and unseen distractor cubes (blue and magenta) near the target cube (red); and (3) Scenario 2 with additional changes in the texture of the table and box, and different size of the table. Our policy attains more than 96% ASR in all three unseen scenarios for all the tasks (see Table 2). This illustrates the robustness of our proposed framework in learning transferable visual policies, robust to unseen distractors in a sample-efficient manner.

Conclusion
In this paper, we introduce a two-step distillation method for learning manipulation tasks in obstructed environments in the visual domain. In step one, we use a MP-augmented RL policy as the state-based expert and subsequently learn a visual BC agent from it, removing the motion planner dependency. In step two, we learn a vision-based agent using an asymmetric actor-critic framework. This step is further expedited via proper weight initialization, BC trajectory-guided RL training, and entropy coefficient tuning, making our method highly sample-efficient. Our visual policy combined with domain randomization demonstrates successful zero-shot transfer to unseen environments with new visual domains and distractors. Beyond zero-shot transfer, fine-tuning the policy in the real world by learning a vision-based critic or applying simulation-to-real techniques [36] is our definite future work to realize simulation-to-real transfer.

A Additional Ablation Study
We discuss further ablation results regarding: (1) The effect of the entropy coefficient α on the performance as well as convergence speed for learning our visual policy. (2) The effect of using our proposed weight initialization strategy for the asymmetric actor-critic framework. (3) The effect of behavioral cloning for trajectory smoothing over the motion planner augmented RL trajectories.
(4) The dependency of our method on the success of MoPA-RL.
A.1 Effect of varying the entropy coefficient α In Section Section 3.2, we discuss the role of tuning the entropy coefficient α for maintaining the trade-off between entropy maximization as well as reward maximization for the SAC objective. For our visual policy learning using asymmetric actor-critic framework, we mainly focus on the exploration achieved by our state-based agent using MoPA-RL [1]. This helps us leverage our state agent's experience for providing guided exploration rather than exploring the entire state space again for visual policy learning. Therefore, we focus on the reward maximization objective by choosing a very small value of α. Table 3 reports the log(α) values for our learned state-agent using MoPA-RL. For accelerating our visual policy learning, we use smaller α values to further emphasize on reward maximization. In Figure 5, we show that using smaller values of α assists in faster convergence and improving sample efficiency as compared to using higher α values. Moreover, smaller α values consistently converge for all the environments whereas for larger α values, we can see unstable performance varying across different tasks. We also observe that large values of α undergo large perturbations during training making learning unstable whereas smaller values of α remain stable throughout training, as shown in Figure 6.

A.2 Effect of weight initialization
As discussed in Section Section 4.3, we note that our weight initialization technique for the actor-critic networks before visual policy learning significantly aids in achieving optimal performance and also facilitates improving learning speed. In Figure 7, we see that without appropriate initialization for the asymmetric critic using our state agent and the visual actor using behavioral cloning, the agent achieves minimal or no success rate across all environments up till 1.2M environment steps. On the other hand, our method with initialization converges to a success rate of ≈ 1.0 much earlier than 1.0M environment steps for all three tasks. : Learning curves comparing our method with and without actor-critic initialization for all three environments. Without initialization, the agent achieves minimal or no success rate, whereas it quickly learns to achieve optimal performance with weight initialization across all the tasks.

A.3 Behavioral cloning for trajectory smoothing
As mentioned in Section 3.2.1, we use BC to smooth out jittery motion planner augmented trajectories and store them in an augmented expert replay buffer R e . In Figure 8, we qualitatively compare BC and MoPA-RL's random rollouts with the same start and goal positions. As we can visualize, the BC trajectory path is smoother compared to the trajectory obtained via the MoPA-RL policy. This is indeed significant for a refined transition from a state-based policy to a visual policy in terms of performance as well as convergence speed and sample efficiency (see Figure 3).   Table 4: Average success rate (ASR), episode length (AEL), and discounted return (ADR) of our method compared with MoPA-RL and BC using sub-optimal and optimal MoPA-RL models. Even with sub-optimal training, when our state-based agent does not learn to completely solve the task, our method for distillation into a visual policy achieves high performance and successfully solves the task in lesser steps (smaller AEL).

A.4 Sub-optimal training for MoPA-RL
To analyze the dependence of our approach on the success of the MoPA-RL training in stage one, we experiment with sub-optimal models of MoPA-RL. We train on lesser environment interactions and then use the critic for weight initialization, followed by our asymmetric visual agent training. We train for the same number of environment steps as Table 1 results, i.e., 2M environment steps. We report the results in Table 4a and Table 4b for Sawyer Push and Sawyer Lift tasks, respectively.
As reported in Table 4a and Table 4b, we observe that although our method benefits from initialization using the state-based agent compared to random initialization, it does not heavily depend on the state-based agent's success. This shows that even when MoPA-RL cannot completely solve the task, our framework for distilling the state-based policy into the visual policy achieves optimal performance.

B.1 Network architecture and hyperparameter selection
For training our state-based agent in step one using MoPA-RL [1], we use three layered fully connected neural networks with 256 hidden units and ReLU activation for actor π φ and critic Q ψ networks and MP implementation similar to [1] using the RRT-Connect algorithm. For our asymmetric agent's critic, we retain the critic network same as Q ψ and the visual actor π θ is a 3-layered convolution neural network followed by three fully connected layers with 256 hidden units and Leaky ReLU activation and another set of fully connected layers outputting the mean and the standard deviation of the Gaussian distribution over an action space. The behavioral cloning agent shares the network architecture with the asymmetric visual actor π θ . We specify other hyperparameter details for SAC and behavioral cloning training in Table 5, Table 6 and Table 7.  For behavioral cloning training, we use validation success rate for choosing the optimal model weights rather than using loss on the test dataset. As seen in the Figure 9, we see that a lower error on the test set does not necessarily provide the best validation accuracy. Therefore, we pick the model on the first such epoch which receives 100% validation accuracy. For measuring the validation accuracy after each training epoch, we use 5 episodes each on 6 randomly chosen seeds. We train our BC agent for a total of 140 epochs and select the earliest model with the best validation accuracy for BC trajectory smoothing as well as asymmetric visual actor's weight initialization. We also report the training time and number of epochs (for optimal convergence) required to account for the additional steps required in learning the BC agent in Table 8.

C Domain Randomization
For domain randomization, we randomly sample a variation for the texture, color and lighting conditions for each episode during training. An illustration of the domain randomization samples for   each environment is shown in Figure 10. Training our method with domain randomized environments helps in learning robust policies that are capable of transfer to unseen domains and distractor objects as shown in Section Figure 4.5.

D Environment Details
We simulate all of our 3D environments using MuJoCo physics engine [31]. Subsequently, we describe the observation details for all three environments. We maintain all other specifics regarding the reward functions, success criteria as well as environment intial states same as explained in Yamada et al. [1].

D.1 Sawyer Push
The task is to reach an object placed inside a box and push it towards the goal region using a Sawyer Arm. We define the positions of the goal, object and the end-effector as p goal , p obj and p eef respectively.
Observations. Each state observation s t comprises of the joint state (sin θ, cos θ), angular joint velocity, quaternion end-effector coordinates, p eef the position of the object, the position of the goal p goal , distance between the end-effector and the object, and the distance between the object and goal position. This acts as an input to the critic in the asymmetric framework.
The visual observation o t comprises of the simulated image corresponding to s t and the robot joint space information. This is used as an input to the actor in the asymmetric framework for learning the visual policy.

D.2 Sawyer Lift
For Sawyer Lift, the task is to reach an object placed inside a box, grasp it using the gripper hand and then lift it up above the box height.
Observations. The state observation s t comprises of the joint state (sin θ, cos θ), angular joint velocity, the position of the object position and quaternion, end-effector coordinates, p eef , the position of the goal p goal , and distance between the end-effector and the object. This acts as an input to the critic in the asymmetric framework.
For an input to the asymmetric actor for visual policy learning, the visual observation o t comprises of the simulated image corresponding to s t and the robot joint space information.

D.3 Sawyer Assembly
In Sawyer Assembly, the task is to assemble the fourth leg of a tabletop in its gripper to its corresponding hole where the other three legs are already in place. This needs to be done while avoiding collisions with the obstructions posed by the other three table legs that are already assembled since the table is free to move under collisions.
Observations. The state observation s t comprises of the joint state (sin θ, cos θ), angular velocity, the position of the hole, the head and tail positions of the leg in the gripper hand and its quaternion.
The input to the asymmetric actor for visual policy learning comprises of the simulated image o t and the robot joint state information.