{'fraction_non_alphanumeric': 0.034334209279283164, 'fraction_numerical': 0.01042519277990781, 'mean_word_length': 4.629562265066085, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

End-to-end approaches to autonomous driving commonly rely on expert demonstrations. Although humans are good drivers, they are not good coaches for end-to-end algorithms that demand dense on-policy supervision. On the contrary, automated experts that leverage privileged information can efficiently generate large scale on-policy and off-policy demonstrations. However, existing automated experts for urban driving make heavy use of hand-crafted rules and perform suboptimally even on driving simulators, where ground-truth information is available. To address these issues, we train a reinforcement learning expert that maps bird's-eye view images to continuous low-level actions. While setting a new performance upper-bound on CARLA, our expert is also a better coach that provides informative supervision signals for imitation learning agents to learn from. Supervised by our reinforcement learning coach, a baseline end-to-end agent with monocular camera-input achieves expert-level performance. Our end-to-end agent achieves a 78% success rate while generalizing to a new town and new weather on the NoCrash-dense benchmark and state-of-the-art performance on the challenging public routes of the CARLA LeaderBoard.


Introduction
Even though nowadays, most autonomous driving (AD) stacks [31,49] use individual modules for perception, planning and control, end-to-end approaches have been proposed since the 80's [36] and the success of deep learning brought them back into the research spotlight [5,51]. Numerous works have studied different network architectures for this task [3,17,53], yet most of these approaches use supervised learning with expert demonstrations, which is known to suffer from covariate shift [37,41]. While data augmentation based on view synthesis [2,5,36] can partially alleviate this issue, in this paper, we tackle the problem from the perspective of expert demonstrations.
Expert demonstrations are critical for end-to-end AD algorithms. While imitation learning (IL) methods directly mimic the experts' behavior [3,11], reinforcement learning (RL) methods often use expert demonstrations to improve sample efficiency by pre-training part of the model via supervised learning [28,48]. In general, expert demonstrations can be divided into two categories: (i) Off-policy, where the expert directly controls the system, and the state/observation distribution follows the expert. Off-policy data for AD includes, for example, public driving datasets [7,23,52]. (ii) On-policy, where the system is controlled by the desired agent and the expert "labels" the data. In this case, the state/observation distribution follows the agent, but expert demonstrations are accessible. On-policy data is fundamental to alleviate covariate shift as it allows the agent to learn from its own mistakes, which the expert in the off-policy data does not exhibit. However, collecting adequate on-policy demonstrations from humans is non-trivial. While trajectories and actions taken by the human expert can be directly recorded during off-policy data collection, labeling these targets given sensor measurements turns out to be a challenging task for humans. In practice, only sparse events like human interventions are recorded, which, due to the limited information it contains, is hard to use for training and better suited for RL [2,24,25] than for IL methods.
In this work we focus on automated experts, which in contrast to human experts can generate large-scale datasets with dense labels regardless of whether they are on-policy or off-policy. To achieve expert-level performance, automated experts may rely on exhaustive computations, expensive sensors or even ground truth information, so it is undesirable to deploy them directly. Even though some IL methods do not require on-policy labeling, such as GAIL [21] and inverse RL [1], these methods are not efficient in terms of on-policy interactions with the environment. On the contrary, automated experts can reduce the expensive on-policy interactions. This allows IL to successfully apply automated experts to different aspects of AD. As a real-world example, Pan et al. [35] demonstrated endto-end off-road racing with a monocular camera by imitating a model predictive control expert with access to expensive sensors. In the context of urban driving, [37] showed that a similar concept can be applied to the driving simulator CARLA [13]. Driving simulators are an ideal proving ground for such approaches since they are inherently safe and can provide ground truth states. However, there are two caveats. The first regards the "expert" in CARLA, commonly referred to as the Autopilot (or the roaming agent). The Autopilot has access to ground truth simulation states, but due to the use of hand-crafted rules, its driving skills are not comparable to a human expert's. Secondly, the supervision offered by most automated experts is not informative. In fact, the IL problem can be seen as a knowledge transfer problem and just learning from expert actions is inefficient.
To tackle both drawbacks and motivated by the success of model-free RL in Atari games [19] and continuous control [15], we propose Roach (RL coach), an RL expert that maps bird's-eye view (BEV) images to continuous actions ( Fig. 1 bottom). After training from scratch for 10M steps, Roach sets the new performance upper-bound on CARLA by outperforming the Autopilot. We then train IL agents and investigate more effective training techniques when learning from our Roach expert. Given that Roach uses a neural network policy, it serves as a better coach for IL agents also based on neural networks. Roach offers numerous informative targets for IL agents to learn from, which go far beyond deterministic action provided by other experts. Here we demonstrate the effectiveness of using action distributions, value estimations and latent features as supervisions. Fig. 1 shows the scheme of learning from on-policy supervisions labeled by Roach on CARLA. We also record off-policy data from Roach by using its output to drive the vehicle on CARLA. Leveraging 3D detection algorithms [27,50] and extra sensors to synthesize the BEV, Roach could also address the scarcity of on-policy supervisions in the real world. This is feasible because on the one hand, BEV as a strong abstraction reduces the sim-to-real gap [32], and on the other hand, on-policy labeling does not have to happen in real-time or even onboard. Hence 3D de-tection becomes easier given the complete sequences [38].
In summary, this paper presents Roach, an RL expert that sets a new performance upper-bound on CARLA. Moreover, we demonstrate the state-of-the-art performance on both the NoCrash benchmark and the public routes of CARLA LeaderBoard using a single camera based endto-end IL agent, which is supervised by Roach using our improved training scheme. Our repository is available at https://github.com/zhejz/carla-roach

Related Work
Since our methods are trained and evaluated on CARLA, we mainly focus on related works also done on CARLA.
End-to-End IL: Dosovitskiy et al. [13] introduced the CARLA driving simulator and demonstrated that a baseline end-to-end IL method with single camera input can achieve a performance comparable to a modular pipeline. After that, CIL [11] and CILRS [12] addressed directional multimodality in AD by using branched action heads where the branch is selected by a high-level directional command. While the aforementioned methods are trained via behavior cloning, DA-RB [37] applied DAGGER [41] with critical state sampling to CILRS. Most recently, LSD [33] increased the model capacity of CILRS by learning a mixture of experts and refining the mixture coefficients using evolutionary optimization. Here, we use DA-RB as the baseline IL agent to be supervised by Roach.
Mid-to-X IL: Directly mapping camera images to low-level actions requires a large amount of data, especially if one wants generalization to diverse weather conditions. Mid-to-X approaches alleviate this issue by using more structured intermediate representation as input and/or output. CILRS with coarse segmentation masks as input was studied in [4]. CAL [42] combines CIL and direct perception [8] by mapping camera images to driving affordances which can be directly used by a rule-based low-level controller. LBC [9] maps camera images to waypoints by mimicking a privileged mid-to-mid IL agent similar to Chauffeurnet [3], which takes BEV as input and outputs future waypoints. Similarly, SAM [54] trained a visuomotor agent by imitating a privileged CILRS agent that takes segmentation and affordances as inputs. Our Roach adopts BEV as the input representation and predicts continuous low-level actions.

RL:
As the first RL agent on CARLA, an A3C agent [30] was demonstrated in [13], yet its performance is lower than that of other methods presented in the same paper. CIRL [28] proposed an end-to-end DDPG [29] agent with its actor network pre-trained via behavior cloning to accelerate online training. To reduce the problem complexity, Chen et al. [10] investigated DDQN [16], TD3 [14] and SAC [15] using BEV as an input and pre-trained the image encoder with a variational auto-encoder [26] on expert tra-jectories. State-of-the-art performance is achieved in [48] using Rainbow-IQN [47]. To reduce the number of trainable parameters during online training, its image encoder is pre-trained to predict segmentation and affordances on an off-policy dataset. IL was combined with RL in [40] and multi-agent RL on CARLA was discussed in [34]. In contrast to these RL methods, Roach achieves high sample efficiency without using any expert demonstrations.
IL with Automated Experts: The effectiveness of automated experts was demonstrated in [35] for real-world offroad racing, where a visuomotor agent is trained by imitating on-policy actions labeled by a model predictive control expert equipped with expensive sensors. Although CARLA already comes with the Autopilot, it is still beneficial to train a proxy expert based on deep neural networks, as shown by LBC [9] and SAM [54]. Through a proxy expert, the complex to solve end-to-end problem is decomposed into two simpler stages. At the first stage, training the proxy expert is made easier by formulating a mid-to-X IL problem that separates perception from planning. At the second stage, the end-to-end IL agent can learn more effectively from the proxy expert given the informative targets it supplies. To provide strong supervision signals, LBC queries all branches of the proxy expert and backpropagates all branches of the IL agent given one data sample, whereas SAM matches latent features of the proxy expert and the end-to-end IL agent. While the proxy expert addresses planning, it is also possible to address perception at the first stage as shown by FM-Net [22]. Overall, twostage approaches achieve better performance than direct IL, but using proxy experts inevitably lowers the performance upper-bound as a proxy expert trained via IL cannot outperform the expert it imitates. This is not a problem for Roach, which is trained via RL and outperforms the Autopilot.

Method
In this section we describe Roach and how IL agents can benefit from diverse supervisions supplied by Roach.

RL Coach
Our Roach has three features. Firstly, in contrast to previous RL agents, Roach does not depend on data from other experts. Secondly, unlike the rule-based Autopilot, Roach is end-to-end trainable, hence it can generalize to new scenarios with minor engineering efforts. Thirdly, it has a high sample efficiency. Using our proposed input/output representation and exploration loss, training Roach from scratch to achieve top expert performance on the six LeaderBoard maps takes less than a week on a single GPU machine.
Roach consists of a policy network π θ (a|i RL , m RL ) parameterized by θ and a value network V φ (i RL , m RL ) parameterized by φ. The policy network maps a BEV image i RL and a measurement vector m RL to a distribution of actions a. Finally the value network estimates a scalar value v, while taking the same inputs as the policy network.
Input Representation: We use a BEV semantic segmentation image i RL ∈ [0, 1] W ×H×C to reduce the problem complexity, similar to the one used in [3,9,10]. It is rendered using ground-truth simulation states and consists of C grayscale images of size W × H. The ego-vehicle is heading upwards and is centered in all images at D pixels above the bottom, but it is not rendered. Fig. 2 illustrates each channel of i RL . Drivable areas and intended routes are rendered respectively in Fig. 2a and 2b. In Fig. 2c solid lines are white and broken lines are grey. Fig. 2d is a temporal sequence of K grayscale images in which cyclists and vehicles are rendered as white bounding boxes. Fig. 2e is the same as Fig. 2d but for pedestrians. Similarly, stop lines at traffic lights and trigger areas of stop signs are rendered in Fig. 2f. Red lights and stop signs are colored by the brightest level, yellow lights by an intermediate level and green lights by a darker level. A stop sign is rendered if it is active, i.e. the ego-vehicle enters its vicinity and disappears once the ego-vehicle has made a full stop. By letting the BEV representation memorize if the ego-vehicle has stopped, we can use a network architecture without recurrent structure and hence reduce the model size of Roach. A colored combination of all channels is visualized in Fig.  1. We also feed Roach a measurement vector m RL ∈ R 6 containing the states of the ego-vehicle not represented in the BEV, these include ground-truth measurements of steering, throttle, brake, gear, lateral and horizontal speed. An effective way to reduce the problem complexity is predicting waypoint plans which are then tracked by a PIDcontroller to produce low-level actions [9,40]. However, a PID-controller is not reliable for trajectory tracking and requires excessive parameter tuning. A model-based controller would be a better solution, but CARLA's vehicle dynamics model is not directly accessible. To avoid parameter tuning and system identification, Roach directly predicts action distributions. Its action space is a ∈ [−1, 1] 2 for steering and acceleration, where positive acceleration corresponds to throttle and negative corresponds to brake. To describe actions we use the Beta distribution B(α, β), where α, β > 0 are respectively the concentration on 1 and 0. Compared to the Gaussian distribution, which is commonly used in model-free RL, the support of the Beta distribution is bounded, thus avoiding clipping or squashing to enforce input constraints. This results in a better behaved learning problem since no tanh layers are needed and the entropy and KL-divergence can be computed explicitly. Further, the modality of the Beta distribution is also suited for driving, where extreme maneuvers may often be taken, for example, emergency braking or taking a sharp turn.
Training: We use proximal policy optimization (PPO) [44] with clipping to train the policy network π θ and the value network V φ . To update both networks, we collect trajectories by executing π θ k on CARLA. A trajectory τ = {(i RL,k , m RL,k , a k , r k ) T k=0 , z} includes BEV images i RL , measurement vectors m RL , actions a, rewards r and a terminal event z ∈ Z that triggers the termination of an episode. The value network is trained to regress the expected returns, whereas the policy network is updated via The first objective L ppo is the clipped policy gradient loss with advantages estimated using generalized advantage estimation [43]. The second objective L ent is a maximum entropy loss commonly employed to encourage exploration Intuitively L ent pushes the action distribution towards a uniform prior because maximizing entropy is equivalent to minimizing the KL-divergence to a uniform distribution, if both distributions share the same support. This inspires us to propose a generalized form of L ent , which encourages exploration in sensible directions that comply with basic traffic rules. We call it the exploration loss and define it as where 1 is the indicator function and z ∈ Z is the event that ends the episode. The terminal condition set Z includes collision, running traffic light/sign, route deviation and being blocked. Unlike L ent which imposes a uniform prior on the actions at all time steps regardless of which z is triggered, L exp shifts actions within the last N z steps of an episode towards a predefined exploration prior p z which encodes an "advice" to prevent the triggered event z from happening again. In practice, we use N z = 100, ∀z ∈ Z. If z is related to a collision or running traffic light/sign, we apply p z = B(1, 2.5) on the acceleration to encourage Roach to slow down while the steering is unaffected. In contrast, if the car is blocked we use an acceleration prior B(2.5, 1). For route deviations, a uniform prior B(1, 1) is applied on the steering. Despite being equivalent to maximizing entropy in this case, the exploration loss further encourages exploration on steering angles during the last 10 seconds before the route deviation.
Implementation Details: Our implementation of PPO-clip is based on [39] and the network architecture is illustrated in Fig. 3a. We use six convolutional layers to encode the BEV and two fully-connected (FC) layers to encode the measurement vector. Outputs of both encoders are concatenated and then processed by another two FC layers to produce a latent feature j RL , which is then fed into a value head and a policy head, each with two FC hidden layers. Trajectories are collected from six CARLA servers at 10 FPS, each server corresponds to one of the six LeaderBoard maps. At the beginning of each episode, a pair of start and target location is randomly selected and the desired route is computed using the A * algorithm. Once the target is reached, a new random target will be chosen, hence the episode is endless unless one of the terminal conditions in Z is met. We use the reward of [47] and additionally penalize large steering changes to prevent oscillating maneuvers. To avoid infractions at high speed, we add an extra penalty proportional to the ego-vehicle's speed. More details are in the supplement.

IL Agents Supervised by Roach
To allow IL agents to benefit from the informative supervisions generated by Roach, we formulate a loss for each of the supervisions. Our training scheme using Roach can be applied to improve the performance of existing IL agents.

CILRS:
The network architecture of CILRS is illustrated in Fig. 3b, it includes a perception module that encodes the camera image i IL and a measurement module that encodes the measurement vector m IL . Outputs of both modules are concatenated and processed by FC layers to generate a bottleneck latent feature j IL . Navigation instructions are given as discrete high-level commands and for each kind of command a branch is constructed. All branches share the same architecture, while each branch contains an action head predicting continuous actions a and a speed head predicting the current speed s of the ego-vehicle. The latent feature j IL is processed by the branch selected by the command. The imitation objective of CILRS consists of an L1 action loss and a speed prediction regularization where λ s is a scalar weight,â is the expert's action,ŝ is the measured speed, a and s are action and speed predicted by CILRS. Expert actionsâ may come from the Autopilot, which directly outputs deterministic actions, or from Roach, where the distribution mode is taken as the deterministic output. Besides deterministic actions, Roach also predicts action distributions, values and latent features. Next we will formulate a loss function for each of them.
Action Distribution Loss: Inspired by [20] which suggests soft targets may provide more information per sample than hard targets, we propose a new action loss based on the action distributions as a replacement for L A . The action head of CILRS is modified to predict distribution parameters and the loss is formulated as a KL-divergence between the action distributionπ predicted by the Roach expert and π predicted by the CILRS agent.
Feature Loss: Feature matching is an effective way to transfer knowledge between networks and its effectiveness in supervising IL driving agents is demonstrated in [22,54]. The latent feature j RL of Roach is a compact representation that contains essential information for driving as it can be mapped to expert actions using an action head consists of only two FC layers (cf. Fig. 3a). Moreover, j RL is invariant to rendering and weather as Roach uses the BEV representation. Learning to embed camera images to the latent space of j RL should help IL agents to generalize to new weather and new situations. Hence, we propose the feature loss Value Loss: Multi-task learning with driving-related side tasks could also boost the performance of end-to-end IL driving agents as shown in [51], which used scene segmentation as a side task. Intuitively, the value predicted by Roach contains driving relevant information because it estimates the expected future return, which relates to how dangerous a situation is. Therefore, we augment CILRS with a value head and regress value as a side task. The value loss is the mean squared error betweenv, the value estimated by Roach, and v, the value predicted by CILRS, Implementation Details: Our implementation follows DA-RB [37]. We choose a Resnet-34 pretrained on ImageNet as the image encoder to generate a 1000-dimensional feature given i RL ∈ [0, 1] 900×256×3 , a wide-angle camera image with a 100 • horizontal FOV. Outputs of the image and the measurement encoder are concatenated and processed by three FC layers to generate j IL ∈ R 256 , which shares the same size as j RL . More details are found in the supplement.

Experiments
Benchmarks: All evaluations are completed on CARLA 0.9.11. We evaluate our methods on the NoCrash [12] and the offline LeaderBoard benchmark 1 [46]. Each benchmark specifies its training towns and weather, where the agent is allowed to collect data, and evaluates the agent in new towns and weather. The NoCrash benchmark considers generalization from Town 1, a European town composed of solely one-lane roads and T-junctions, to Town 2, a smaller version of Town 1 with different textures. By contrast, the LeaderBoard considers a more difficult generalization task in six maps that cover diverse traffic situations, including freeways, US-style junctions, roundabouts, stop signs, lane changing and merging. Following the NoCrash benchmark, we test generalization from four training weather types to two new weather types. But to save computational resources, only two out of the four training weather types are evaluated. The NoCrash benchmark comes with three levels of traffic density (empty, regular and dense), which defines the number of pedestrians and vehicles in each map. We focus on the NoCrash-dense and introduce a new level between regular and dense traffic, NoCrash-busy, to avoid congestion that often appears in the dense traffic setting. For the offline LeaderBoard the traffic density in each map is tuned to be comparable to the busy traffic setting.
Metrics: Our results are reported in success rate, the metric proposed by NoCrash, and driving score, a new metric introduced by the CARLA LeaderBoard. The success rate is the percentage of routes completed without collision or blockage. The driving score is defined as the product of route completion, the percentage of route distance completed, and infraction penalty, a discount factor that aggregates all triggered infractions. For example, if the agent ran two red lights in one route and the penalty coefficient for running one red light was 0.7, then the infraction penalty would be 0.7 2 =0.49. Compared to the success rate, the driving score is a fine-grained metric that considers more kinds of infractions and it is better suited to evaluate longdistance routes. More details about the benchmarks and the complete results are found in the supplement.

Performance of Experts
We use CARLA 0.9.10.1 to train RL experts and finetune our Autopilot, yet all evaluations are still on 0.9.11.
Sample Efficiency: To improve the sample efficiency of PPO, we propose to use BEV instead of camera images, Beta instead of Gaussian distributions, and the exploration loss in addition to the entropy loss. Since the benefit of using a BEV representation is obvious, here we only ablate the Beta distribution and the exploration loss. As shown in Fig. 4, the baseline PPO with Gaussian distribution and entropy loss is trapped in a local minimum where staying still is the most rewarding strategy. Leveraging the exploration loss, PPO+exp can be successfully trained despite relatively high variance and low sample efficiency. The Beta distribution helps substantially, but without the exploration loss the training still collapsed in some cases due to insufficient exploration (cf. dashed blue line in Fig. 4). Our Roach (PPO+beta+exp) uses both Beta distribution and exploration loss to ensure stable and sample efficient training. The training takes around 1.7M steps in each of the six CARLA servers, this accounts for 10M steps in total, which takes roughly a week on an AWS EC2 g4dn.4xlarge or 4 days on a 2080 Ti machine with 12 cores.
Driving Performance: Table 1 compares different experts on the NoCrash-dense and on all 76 LeaderBoard routes under dynamic weather with busy traffic. Our Autopilot is a strong baseline expert that achieves a higher success rate than the Autopilot used in LBC and DA-RB. We evaluate three RL experts -(1) Roach, the proposed RL coach using Beta distribution and exploration prior. (2) PPO+beta, the RL coach trained without using the exploration prior. (3) PPO+exp, the RL coach trained without using the Beta distribution. In general, our RL experts achieve comparable success rates and higher driving scores than Autopilots because RL experts handle traffic lights in a better way (cf. Table 3). The two Autopilots often run red lights because they drive over-conservatively and wait too long at the junction, thus missing the green light. Among RL experts, PPO+beta and Roach, the two RL experts using a Beta distribution, achieve the best performance, while the difference between both is not significant. PPO+exp performs slightly worse, but it still achieves better driving scores than our Autopilot.

Performance of IL Agents
The performance of an IL agent is limited by the performance of the expert it is imitating. If the expert performs poorly, it is not sensible to compare IL agents imitating that expert. As shown in Table 1, this issue is evident in the NoCrash new town with dense traffic, where Autopilots do not perform well. To ensure a high performance upperbound and hence a fair comparison, we conduct ablation studies ( Fig. 5 and Table 3) under the busy traffic setting such that our Autopilot can achieve a driving score of 80% and a success rate of 90%. In order to compare with the state-of-the-art, the best model from the ablation studies is still evaluated on NoCrash with dense traffic in Table 2.
route is specified as sparse GPS locations. The LeaderBoard instruction suggests that it is used to disambiguate situations where the semantics of left and right are not clear due to the complexity of the considered map.
Ablation: Fig. 5 shows driving scores of experts and IL agents at each DAGGER iteration on NoCrash and offline LeaderBoard with busy traffic. The baseline L A (AP) is our implementation of DA-RB + supervised by our Autopilot. Given our improved Autopilot, it is expected that L A (AP) can achieve higher success rates than those reported in the DA-RB paper, but this is not observed in Table 2. The large performance gap between the Autopilot and L A (AP) (cf. Fig. 5), especially while generalizing to a new town and new weather, indicates the limitation of this baseline.
By replacing the Autopilot with Roach, L A performs bet-ter overall than L A (AP). Further learning from the action distribution, L K generalizes better than L A on the NoCrash but not on the offline LeaderBoard. Feature matching only helps when j IL is provided with the necessary information needed to reproduce j RL . In our case, j RL contains navigational information as the desired route is rendered in the BEV input. For the LeaderBoard, navigational information is partially encoded in m IL , which includes the vector to the next desired waypoint, so better performance is observed by using L F . But for NoCrash this information is missing as m IL is just the speed, hence it is impractical for j IL to mimic j RL and this causes the inferior performance of L K + L F and L K + L F + L V . To confirm this hypothesis, we evaluate a single-branch network architecture where the measurement vector m IL is augmented by the command encoded as a onehot vector. Using feature matching with this architecture, L K + L F (c) and L K + L V + L F (c) achieve the best driving score among IL agents in the NoCrash new town & weather generalization test, even outperforming the Autopilot.
Using value supervision in addition to feature matching helps the DAGGER process to converge faster as shown by L K + L V + L F and L K + L V + L F (c). However, without feature matching, using value supervision alone L K + L V does not demonstrate superior performance. This indicates a potential synergy between feature matching and value estimation. Intuitively, the latent feature of Roach encodes the information needed for value estimation, hence mimicking this feature should help to predict the value, while value estimation could help to regularize feature matching.
Comparison with the State-of-the-art: In Table 2 we compare the baseline L A (AP) and our best performing agent L K + L F (c) with the state-of-the-art on the NoCrashdense benchmark. Our L A (AP) performs comparably to DA-RB + except when generalizing to the new weather, where there is an incorrect rendering of after-rain puddles on CARLA 0.9.11 (see supplement for visualizations).This issue does not affect our best method L K + L F (c) due to the stronger supervision of Roach. By mimicking the weatheragnostic Roach, the performance of our IL agent drops by less than 10% while generalizing to the new town and weather. Hence if the Autopilot is considered the performance upper-bound, it is fair to claim our approach saturates the NoCrash benchmark. However, as shown in Fig. 5, there is still space for improvement on NoCrash compared to Roach and the performance gap on the offline Leader-Board highlights the importance of this new benchmark.

Roach
95 ± 2 96 ± 3 100 ± 0 96 ± 3 0 ± 0 0.11 ± 0.07 0.04 ± 0.05 0.16 ± 0.20 0 ± 0 Autopilot 91 ± 1 79 ± 2 98 ± 1 80 ± 2 0 ± 0 0 ± 0 0.18 ± 0.08 1.93 ± 0.23 0.18 ± 0.08 itating Roach, which drives more naturally, and L A shows an absolute improvement of 23% in terms of driving score. In other words this is the gain achieved by using a better expert, but the same imitation learning approach. Further using the improved supervision from soft targets and latent features results in our best model L K +L F (c), which demonstrates another 22% absolute improvement. By handling red lights in a better way, this agent achieves 88%, an expertlevel driving score, using a single camera image as input.

Conclusion
We present Roach, an RL expert, and an effective way to imitate this expert. Using the BEV representation, Beta distribution and the exploration loss, Roach sets the new performance upper-bound on CARLA while demonstrating high sample efficiency. To enable a more effective imitation, we propose to learn from soft targets, values and latent features generated by Roach. Supervised by these informative targets, a baseline end-to-end IL agent using a single camera image as input can achieve state-of-the-art performance, even reaching expert-level performance on the NoCrash-dense benchmark. Future works include performance improvement on simulation benchmarks and realworld deployment. To saturate the LeaderBoard, the model capacity shall be increased [3,18,33]. To apply Roach to label real-world on-policy data, several sim-to-real gaps have to be addressed besides the photorealism, which is partially alleviated by the BEV. For urban driving simulators, the realistic behavior of road users is of utmost importance [45].

A. Summary
In the appendix, we provide (1) an overview of supplementary videos and codes, (2) implementation details of the RL experts and the IL agents, (3) details regarding benchmarks, and (4) additional experimental results.

B.1. Videos
To investigate how different agents actually drive, we provide three videos. roach.mp4 shows the driving performance of Roach, and highlights that it has a natural driving style and that it can handle complex traffic scenes. In autopilot.mp4 we demonstrate the rule-based CARLA Autopilot. This agent uses unnatural brake actuation, i.e. it only uses emergency braking. Further, this video also highlights that in dense traffic, the rule-based agent can get stuck due to conservative danger predictions. For more details about the Autopilot and changes we made see Section C.3. Finally, in il agent.mp4 we demonstrate our best roachsupervised IL agent, showing that the agent can handle complex traffic scenes but also highlighting failure cases. In detail: • roach.mp4 is an uncut evaluation run recorded from Roach driving in Town03 (LeaderBoard-busy under dynamic weather). This video demonstrates the natural driving style of Roach even in challenging situations such as US-style traffic lights, unprotected left turns, roundabouts and stop signs.
• autopilot.mp4 is an uncut evaluation run recorded from Autopilot driving in Town02 (NoCrash-dense, new town & new weather). This video demonstrates the over-conservative behavior of the Autopilot while driving in dense traffic. This often leads to red light infractions and blockage (both are present in the video).
• il agent.mp4 is a highlight video recorded from our best roach-supervised IL agent L K +L F (c). This video includes multiple challenging situations often encountered during urban driving, such as EU and US-style junctions, unprotected left turns, roundabouts and reacting to pedestrians walking into the street. Furthermore, we highlight some of the failure modes of our camera-based IL agent, including not coming to a full stop for stop signs, collisions at overcrowded intersections and oscillation in the steering if the lane markings are not visible due to sun glare. We believe that including memory in the IL agent policy can help in most of these issues, due to a better understanding of the egomotion (stop sign and oscillations) and other agents' motion (collisions).

B.2. Code
To reproduce our results, we provide four python scripts: • train rl.py for training Roach.
• data collect.py for collecting on/off-policy data.
It is recommended to run our scripts through bash files contained in the folder run. All configurations are in the folder config. Our repository is composed of two modules: • carla gym, a versatile OpenAI gym [6] environment for CARLA. It allows not only RL training with synchronized rollouts, but also data collection and evaluation. The environment is configurable in terms of weather, number of background pedestrians and vehicles, benchmarks, terminal conditions, sensors, rewards for the ego-vehicle and etc.

B.3. Rendering issues
As illustrated in Fig. 6, on CARLA 0.9.11 reflections from after-rain puddles are sometimes wrongly rendered as black pixels. When the black pixels are accumulated, for example in the middle of Fig. 6a, they are often recognized as obstacles by the camera-based agents. Since this kind of reflection only appears under the testing weather but not under the training weather, generalizing to testing weather is exceptionally hard on CARLA 0.9.11 for the camera-based end-to-end IL agents.

C.1. Roach
The network architecture of Roach can be found in Table 6 and the hyper-parameter values are listed in Table 8.

BEV:
Cyclists and pedestrians are rendered larger than their actual sizes, this allows us to use a smaller image encoder with less parameters for Roach. Additionally, increasing the size naturally adds some caution when dealing with these vulnerable road users.
Update: The policy network and the value network are updated together using one Adam optimizer with an initial learning rate of 1e-5. The learning rate is scheduled based on the empirical KL-divergence between the policy before and after the update. If the KL-divergence is too large after an update epoch, the update phase will be interrupted  and a new rollout phase will start. Furthermore, a patience counter will be increased by one and the learning rate will be reduced once the patience counter reaches a threshold.
Rollout: Before each update phase a fixed-size buffer will be filled with trajectories collected on six CARLA servers, each corresponds to one of the six LeaderBoard maps (Town1-6).
Terminal Condition: An episode is terminated if and only if one of the following event happens. • Route deviation: triggered if the lateral distance to the lane centerline of the desired route is larger than 3.5 meters. Terminal reward: −1.
• Blocked: speed of the ego-vehicle is slower than 0.1 m/s for more than 90 consecutive seconds. Terminal reward: −1.
with s is the ego-vehicle's speed. The terminal reward is the reward given to the very last observation/action pair before the termination. For non-terminal samples, the terminal reward is 0.
Reward Shaping: The reward is the sum of the following components.
• r speed: equals to 1.0 − |s − s desired |/s max , where s is the measured speed of the ego-vehicle, s max is the maximum speed and s desired is the desired speed. We use a constant maximum speed s max = 6 m/s. The desired speed is a variable and is explained below.
• r position: equals to −0.5∆ p , where ∆ p is the lateral distance (in meters) between the ego-vehicle's center and the center line of the desired route.
• r rotation: equals to −∆ r , where ∆ r is the absolute value of the angular difference (in radians) between the ego-vehicle's heading and the heading of the center line of the desired route.
• r action: equals to −0.1 if the current steering differs more than 0.01 from the steering applied in the previous step.
• r terminal: the aforementioned terminal reward.
The desired speed, as proposed in [47], depends on rulebased obstacle detections. If there's no obstacle detected, the desired speed equals to the maximum speed. If an obstacle is detected, based on the distance to the obstacle the desired speed is linearly decreased to 0. As obstacle detector we use the hazard detection of Autopilot (cf. Section C.3). As a dense and informative reward, r speed helps substantially to train our Roach and the camera-based end-to-end RL agent [47]. However, using rule-based obstacle detections inevitable introduces bias, the trained RL agent can be over-aggressive or over-conservative depending on the false positive and false negative rate of the detector. For example, during multi-lane freeway driving, our Roach decelerates for vehicles on the neighbouring lanes because those vehicles are detected as obstacles during training. Another example, Roach tends to collide after a right turn, this is related to the sector shaped (around 40 degrees) detection area used by the obstacle detection; vehicles and pedestrians on the right are not covered in the detection area. To further improve the performance of Roach, this r speed should be modified, either using a better obstacle detector, or completely remove the rule-based obstacle detection, and build a less artificial reward based on simulation states.

Mode of Beta Distribution:
We take the distribution mode as a deterministic output. The mode of the Beta distribution B(α, β) is defined as For a natural driving behavior, we use the mean α α+β as the deterministic output when the mode is not uniquely defined, i.e. when α < 1, β < 1 or α = 1, β = 1.

C.2. IL Agent Supervised by Roach
The network architecture of our IL agent is found in Table 7 and the hyper-parameter values are listed in Table 9. Off-policy Data Collection: Following CILRS [12], triangular perturbations on actions are applied while collecting the off-policy expert dataset to alleviate the covariate shift.
The off-policy dataset for NoCrash includes 80 episodes and for LeaderBoard it includes 160 episodes. Each episode is at most 300 seconds and at least 30 seconds long. The episode will be terminated if the expert violates any traffic rules, including red light infractions, stop sign infractions and collisions. In such a case, we remove the last 30 seconds of that episode so as to ensure that the off-policy dataset includes only correct demonstrations. Data is not collected using the given training routes but from randomly spawned start and target locations.
On-policy Data Collection: We follow DA-RB [37] for DAGGER with critical state sampling and replay buffer. New DAGGER-data will replace the old data in the replay buffer, while the buffer size is fixed. The same number of frames are contained in the replay buffer as in the off-policy dataset. At each DAGGER iteration, around 15-25% of the replay buffer is filled with new DAGGER-data, whereas at least 20% of the replay buffer is filled with off-policy data. Identical to the off-policy data collection, we use randomly spawned start and target locations while collecting DAG-GER datasets. Following DA-RB, we did not use a mixed agent/expert policy to collect DAGGER datasets. However, our code allows this kind of rollout for DAGGER.
Training Details: Since we take the ResNet-34 pre-trained on ImageNet, the input image is normalized as suggested. In case the IL agent uses a distributional action head and/or a value head, the corresponding weights will be loaded from the Roach model at the first training iteration (the behavior cloning iteration). At each DAGGER iteration, the training continuous from the last epoch of the previous DAGGER iteration. We apply image augmentations using code modified from CILRS. The image augmentation methods are applied in random order and include Gaussian blur, additive Gaussian noise, coarse and block-wise dropouts, additive and multiplicative noise to each channel, randomized contrast and grayscale. All models are trained for 25 epochs using the ADAM optimizer with an initial learning rate of 2e-4. The learning rate is halved if the validation loss has not decreased for more than 5 epochs.

C.3. Autopilot
The CARLA Autopilot (also called roaming agent) is a simple but effective automated expert based on hand-crafted rules and ground-truth simulation states. The Autopilot is composed of two PID controllers for trajectory tracking and hazard detectors for emergency brake. Hazards include • pedestrians/vehicles detected ahead, • red lights/stop sings detected ahead, • negative ego-vehicle speed, for handling slopes.
Locations and states of pedestrians, vehicles, red lights and stop signs are provided as ground-truth by the CARLA API. If any hazard appears in a trigger area ahead of the ego-vehicle, Autopilot will make an emergency brake with throttle = 0, steering = 0, brake = 1. If no hazard is detected, the ego-vehicle will follow the desired path using two PID controllers, one for speed and one for steering control. The PID controller takes as input the location, rotation and speed of the ego-vehicle and the desired route specified as dense (1 meter interval) waypoints. The speed PID yields throttle ∈ [0, 1] and the steering PID yields steering ∈ [−1, 1]. We tuned the parameters for PID controllers and hazard detectors manually, such that the Autopilot is a strong baseline. The target speed is 6 m/s.

D. Benchmarks
Scope: The scope of the NoCrash and the offline Leader-Board benchmark are illustrated in Table 4. The offline LeaderBoard benchmark considers more traffic scenarios and longer routes in six different maps.
Weather: Following the NoCrash benchmark, we use ClearNoon, WetNoon, HardRainNoon and ClearSunset as the training weather types, whereas new weather types are SoftRainSunset and WetSunset. To save computational resources, only two out of the four training weather types are evaluated, they are WetNoon and ClearSunset.
Background Traffic: The number of vehicles and pedestrians spawned in each map of different benchmarks are listed in Table 5. Vehicles and pedestrians are spawned randomly from the complete blueprint library of CARLA 0.9.11. This stands in contrast to several previous works where for example two-wheeled vehicles are disabled.
Pros and cons of the online and the offline Leaderboard: Online Leaderboard: (+) All methods are evaluated under exactly the same condition. (+) No need to re-evaluate other methods. (-) No restriction on how the method is trained and how the training data is collected.
Offline Leaderboard: (+) Strictly prescribes both the training and testing environment. (+) Full control and observation over the benchmark. (-) You will have to re-evaluate other methods, if any setup of the benchmark has changed, for example CARLA version and etc.
One can use the offline Leaderboard if a thorough study on the generalization ability of the method is desired.

E. Additional Experimental Results
To verify IL agents trained using the feature loss indeed embed camera images to the latent space of Roach, we report the feature loss at test time in Fig. 7. In the first row of Fig. 7, the IL agent trained without feature loss, L K , learns a latent space independent of the one of Roach. Hence, the test feature loss is effectively noise that is invariant to the test condition. In the second row, L K + L F (c) is trained   with the feature loss. The test feature loss of this agent is much smaller (less than 1) and increases as expected during the generalization tests. To complete Fig. 5 of the main paper, driving scores of experts and IL agents at each DAGGER iterations are in Fig. 8 (NoCrash-busy) and Fig. 9 (LeaderBoard-busy).