{'fraction_non_alphanumeric': 0.03770755222281735, 'fraction_numerical': 0.012283520799857168, 'mean_word_length': 4.795943708609271, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PlanT: Explainable Planning Transformers via Object-Level Representations

Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations containing all vehicle and road context information. In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. PlanT is based on imitation learning with a compact object-level input representation. On the Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3x faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf perception module provides a sensor-based driving system that is more than 10 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.


Introduction
The ability to plan is an important aspect of human intelligence, allowing us to solve complex navigation tasks. For example, to change lanes on a busy highway, a driver must wait for sufficient space in the new lane and adjust the speed based on the expected behavior of the other vehicles. Humans quickly learn this and can generalize to new scenarios, a trait we would also like autonomous agents to have. Due to the difficulty of the planning task, the field of autonomous driving is shifting away from traditional rule-based algorithms [1,2,3,4,5,6,7,8] towards learning-based solutions [9,10,11,12,13,14]. Learning-based planners directly map the environmental state representation (e.g., HD maps and object bounding boxes) to waypoints or vehicle controls. They emerged as a scalable alternative to rule-based planners which require significant manual effort to design.
Interestingly, while humans reason about the world in terms of objects [15,16,17], most existing learned planners [9,12,18] choose a high-dimensional pixel-level input representation by rendering bird's eye view (BEV) images of detailed HD maps (Fig. 1 left). It is widely believed that this kind of accurate scene understanding is key for robust self-driving vehicles, leading to significant interest in recovering pixel-level BEV information from sensor inputs [19,20,21,22,23,24]. In this paper, we investigate whether such detailed representations are actually necessary to achieve convincing planning performance. We propose PlanT, a learning-based planner that leverages an object-level representation ( Fig. 1 right) as an input to a transformer encoder [25]. We represent a scene as a set of features corresponding to (1) nearby vehicles and (2) the route the planner must follow. We show that despite the low feature dimensionality, our model achieves state-of-the-art results. We then propose a novel evaluation scheme and metric to analyze explainability which is generally applicable to any learning-based planner. Specifically, we test the ability of a planner to identify the objects that are the most relevant to account for to plan a collision-free route. We perform a detailed empirical analysis of learning-based planning on the Longest6 benchmark [26] of the CARLA simulator [27]. We first identify the key missing elements in the design of existing learned planners such as their incomplete field of view and sub-optimal dataset and model sizes. We then show the advantages of our proposed transformer architecture, including improvements in performance and significantly faster inference times. Finally, we show that the attention weights of the transformer, which are readily accessible, can be used to represent object relevance. Our qualitative and quantitative results on explainability confirm that PlanT attends to the objects that match our intuition for the relevance of objects for safe driving.

Contributions.
(1) Using a simple object-level representation, we significantly improve upon the previous state of the art for planning on CARLA via PlanT, our novel transformer-based approach.
(2) Through a comprehensive experimental study, we identify that the ego vehicle's route, a full 360°field of view, and information about vehicle speeds are critical elements of a planner's input representation. (3) We propose a protocol and metric for evaluating a planner's prioritization of obstacles in a scene and show that PlanT is more explainable than CNN-based methods, i.e., the attention weights of the transformer identify the most relevant objects more reliably.

Related Work
Intermediate Representations for Driving. Early work on decoupling end-to-end driving into two stages predicts a set of low-dimensional affordances from sensor inputs with CNNs which are then input to a rule-based planner [28]. These affordances are scene-descriptive attributes (e.g. emergency brake, red light, center-line distance, angle) that are compact, yet comprehensive enough to enable simple driving tasks, such as urban driving on the initial version of CARLA [27]. Unfortunately, methods based on affordances perform poorly on subsequent benchmarks in CARLA which involve higher task complexity [29]. Most state-of-the-art driving models instead rely heavily on annotated 2D data either as intermediate representations or auxiliary training objectives [26,30]. Several subsequent studies show that using semantic segmentation as an intermediate representation helps for navigational tasks [31,32,33,34]. More recently, there has been a rapid growth in interest on using BEV semantic segmentation maps as the input representation to planners [9,12,30,18]. To reduce the immense labeling cost of such segmentation methods, Behl et al. [35] propose visual abstractions, which are label-efficient alternatives to dense 2D semantic segmentation maps. They show that reduced class counts and the use of bounding boxes instead of pixel-accurate masks for certain classes is sufficient. Wang et al. [36] explore the use of object-centric representations for planning by explicitly extracting objects and rendering them into a BEV input for a planner. However, so far, the literature lacks a systematic analysis of whether object-centric representations are better or worse than BEV context techniques for planning in dense traffic, which we address in this work. We keep our representation simple and compact by directly considering the set of objects as inputs to our models. In addition to baselines using CNNs to process the object-centric representation, we show that using a transformer leads to improved performance, efficiency, and explainability.
Transformers for Forecasting. Transformers obtain impressive results in several research areas [25,37,38,39], including simple interactive environments such as Atari games [40,41,42,43,44]. While the end objective differs, one application domain that involves similar challenges to planning is motion forecasting. Most existing motion forecasting methods use a rasterized input in combination with a CNN-based network architecture [45,46,47,48,49,50]. Gao et al. [51] show the advantages of object-level representations for motion forecasting via Graph Neural Networks (GNN). Several follow-ups to this work use object-level representations in combination with Transformer-based architectures [52,53,54]. Our key distinctions when compared to these methods are the architectural simplicity of PlanT (our use of simple self-attention transformer blocks and the proposed route representation) as well as our closed-loop evaluation protocol (we evaluate the driving performance in simulation and report online driving metrics).
Explainability. Explaining the decisions of neural networks is a rapidly evolving research field [55,56,57,58,59,60,61]. In the context of self-driving cars, existing work uses text [62] or heatmaps [63] to explain decisions. In our work, we can directly obtain post hoc explanations for decisions of our learning-based PlanT architecture by considering its learned attention. While the concurrent work CAPO [64] uses a similar strategy, it only considers pedestrian-ego interactions on an empty route, while we consider the full planning task in an urban environment with dense traffic. Furthermore, we introduce a simple metric to measure the quality of explanations for a planner.

Planning Transformers
In this section, we provide details about our task setup, novel scene representation, simple but effective architecture, and training strategy resulting in state-of-the-art performance. A PyTorch-style pseudo-code snippet outlining PlanT and its training is provided in the supplementary material.
Task. We consider the task of point-to-point navigation in an urban setting where the goal is to drive from a start to a goal location while reacting to other dynamic agents and following traffic rules. We use Imitation Learning (IL) to train the driving agent. The goal of IL is to learn a policy π that imitates the behavior of an expert π * (the expert implementation is described in Section 4). In our setup, the policy is a mapping π : X − → W from our novel object-level input representation X to the future trajectory W of an expert driver. For following traffic rules, we assume access to the state of the next traffic light relevant to the ego vehicle l ∈ {green, red}.
Tokenization. To encode the task-specific information required from the scene, we represent it using a set of objects, with vehicles and segments of the route each being assigned an oriented bounding box in BEV space ( Fig. 1 right). Let X t = V t ∪S t , where V t ∈ R Vt×A and S t ∈ R St×A represent the set of vehicles and the set of route segments at time-step t with A = 6 attributes each. Specifically, if o i,t ∈ X t represents a particular object, the attributes of o i,t include an object type-specific attribute z i,t (described below), the position of the bounding box (x i,t , y i,t ) relative to the ego vehicle, the orientation ϕ i,t ∈ [0, 2π], and the extent (w i,t , h i,t ). Thus, each object o i,t can be described as a vector a=1 . For the vehicles V t , we extract the attributes directly from the simulator in our main experiments and use an off-the-shelf perception module based on CenterNet [65] (described in the supplementary material) for experiments involving a full driving system. We consider only vehicles up to a distance D max from the ego vehicle, and use o i,t,1 (i.e., z i,t ) to represent the speed.
To obtain the route segments S t , we first sample a dense set of N t points U t ∈ R Nt×2 along the route ahead of the ego vehicle at time-step t. We directly use the ground-truth points from CARLA as U t in our main experiments and predict them with a perception module for the PlanT with perception experiments in Section 4.1. The points are subsampled using the Ramer-Douglas-Peucker algorithm [66,67] to select a subsetÛ t . One segment spans the area between two points subsampled from the route, u i,t , u i+1,t ∈Û t . Specifically, o i,t,1 (i.e., z i,t ) denotes the ordering for the current time-step t, starting from 0 for the segment closest to the ego vehicle. We set the segment length o i,t,6 = ||u i,t − u i+1,t || 2 , and the width, o i,t,5 , equal to the lane width. In addition, we clip o i,t,6 <= L max , ∀i, t; and always input a fixed number of segments N s to our policy. More details and visualizations of the route representation are provided in the supplementary material.
Token Embeddings. Our model is illustrated in Fig. 2. As a first step, applying a transformer backbone requires the generation of embeddings for each input token, for which we define a linear projection ρ : R 6 → R H (where H is the desired hidden dimensionality). To obtain token embeddings e i,t , we add the projected input tokens o i,t to a learnable object type embedding vector e v ∈ R H or e s ∈ R H , indicating to which type the token belongs (vehicle or route segment). We represent a scene (bottom left) using a set of objects containing the vehicles and route to follow (green arrows). We embed these via a linear projection (bottom right) and process them with a transformer encoder. PlanT outputs future waypoints with a GRU decoder. We use a self-supervised auxiliary task of predicting the future of other vehicles. Further, extracting and visualizing the attention weights yields an explainable decision (top left).
Main Task: Waypoint Prediction. The main building block for the IL policy π is a standard transformer encoder, τ : R Vt+St+1×H → R Vt+St+1×H , based on the BERT architecture [37]. Specifically, we define a learnable [CLS] token, c ∈ R H (based on [37,39]) and stack this with other token embeddings to obtain the transformer input [c, e 1,t , ..., e Vt+St,t ]. The [CLS] token's processing through τ involves an attention-based aggregation of the features from all other tokens, after which it is used for generating the waypoint predictions via an auto-regressive waypoint decoder, γ : R (H+1) → R W ×2 . For a detailed description of the waypoint decoder architecture, see [18,26]. We concatenate the binary traffic light flag, l t to the transformer output as the initial hidden state to the decoder which makes use of GRUs [68] to predict the future trajectory W t of the ego vehicle, centered at the coordinate frame of the current time-step t. The trajectory is represented by a sequence of 2D waypoints in BEV space, {w w = (x w , y w )} t+W w=t+1 for W = 4 future time-steps: Auxiliary Task: Vehicle Future Prediction. In addition to the primary waypoint prediction task, we propose the auxiliary task of predicting the future attributes of other vehicles. This is aligned with the overall driving goal in two ways. (1) The ability to reason about the future of other vehicles is important in an urban environment as it heavily influences the ego vehicle's own future.
(2) Our main task is to predict the ego vehicle's future trajectory, which means the output feature of the transformer needs to encode all the information necessary to predict the future. Supervising the outputs of all vehicles on a similar task (i.e., predicting vehicle poses at a future time-step) exploits synergies between the task of the ego vehicle and the other vehicles [69,30]. Specifically, using the output embeddings {h i,t } Vt i=1 corresponding to all vehicle tokens o i,t ∈ V t , we predict class probabilities {{p i,t+1,a } 6 a=1 } Vt i=1 for the speed, position, orientation, and extent attributes from the next time-step {o i,t+1,a } 6 a=1 using a linear layer per attribute type {ψ a : R H → R Za } 6 a=1 : p i,t+1,a = Softmax (ψ a (h i,t )), where a = 1, ..., 6.
(3) We choose to discretize each attribute into Z a bins to allow for uncertainty in the predictions since the future is multi-modal. This is also better aligned with how humans drive without predicting exact locations and velocities, where a rough estimate is sufficient to make a safe decision.
Loss Functions. Following recent driving models [9,11,30,26], we leverage the L 1 loss to the ground truth future waypoints w gt as our main training objective. For the auxiliary task, we calculate the cross-entropy loss L CE using a one-hot encoded representation p gt i,t+1 of the ground truth future vehicle attributes o gt i,t+1 . We train the model in a multi-task setting using a weighted combination of these losses with a weighting factor λ:

Experiments
In this section, we describe our experimental setup, evaluate the driving performance of our approach, analyze the explainability of its driving decisions, and finally discuss limitations.
Dataset and Benchmark. We use the expert, dataset and evaluation benchmark Longest 6 proposed by [26]. The expert policy is a rule-based algorithm with access to ground truth locations of the vehicles as well as privileged information that is not available to PlanT such as their actions and dynamics. Using this information, the expert determines the future position of all vehicles and estimates intersections between its own future position and those of the other vehicles to prevent most collisions. The dataset collected with this expert contains 228k frames. We use this as our reference point denoted by 1×. For our analysis, we also generate additional data following [26] but with different initializations of the traffic. The data quantities we use are always relative to the original dataset (i.e., 2× contains double the data, 3× contains triple). We refer the reader to [26] for a detailed description of the expert algorithm and dataset collection.
Metrics. We report the established metrics of the CARLA leaderboard [70]: Route Completion (RC), Infraction Score (IS), and Driving Score (DS), which is the weighted average of the RC and IS. In addition, we show Collisions with Vehicles per kilometer (CV) and Inference Time (IT) for one forward pass of the model, measured in milliseconds on a single RTX 3080 GPU.
Baselines. To highlight the advantages of learning-based planning, we include a rule-based planning baseline that uses the same inputs as PlanT. It follows the same high-level algorithm as the expert but estimates the future of other vehicles using a constant speed assumption since it does not have access to their actions. AIM-BEV [18] is a recent privileged agent trained using IL. It uses a BEV semantic map input with channels for the road, lane markings, vehicles, and pedestrians, and a GRU identical to PlanT to predict a trajectory for the ego vehicle which is executed using lateral and longitudinal PID controllers. Roach [12] is a Reinforcement Learning (RL) based agent with a similar input representation as AIM-BEV that directly outputs driving actions. Roach and AIM-BEV are the closest existing methods to PlanT. However, they use a different input field of view in their representation leading to sub-optimal performance. We additionally build PlanCNN, a more competitive CNN-based approach for planning with the same training data and input information as PlanT, which is adapted from AIM-BEV to input a rasterized version of our object-level representation. We render the oriented vehicle bounding boxes in one channel, represent the speed of each pixel in a second channel, and render the oriented bounding boxes of the route in the third channel. We provide detailed descriptions of the baselines in the supplementary material.  Table 1: Longest6 Results. We show the mean±std for 3 evaluations. PlanT reaches expert-level performance and requires significantly less inference time than the baselines. *We evaluate the author-provided pre-trained models for LAV and Roach.

Obtaining Expert-Level Driving Performance
In the following, we discuss the key findings of our study which enable expert-level driving with learned planners. We begin with a discussion of the privileged methods and analyze the sensorbased methods in Section 4.1. Unless otherwise specified, the experiments consider the largest version of our dataset (3×) and models (MEDIUM for PlanT, ResNet-34 for PlanCNN).
Input Representation. Table 1 compares the performance on the Longest6 benchmark. The rulebased system acts cautiously and gets blocked often. Among the learning-based methods, both PlanCNN and PlanT significantly outperform AIM-BEV [18] and Roach [12]. We systematically break down the factors leading to this in Table 2a by studying the following: (1) the representation used for the road layout, (2) the horizontal field of view, (3) whether objects behind the ego vehicle are part of the representation, and (4) whether the input representation incorporates speed.
Roach uses the same view to the sides as AIM-BEV but additionally includes 8 m to the back and multiple input frames to reason about speed. We see in Table 2a that  These results show that a full 360°field of view is helpful to handle certain situations encountered during our evaluation (e.g. changing lanes). Finally, removing the vehicle speed input significantly reduces the DS for both PlanCNN and PlanT (Table 2a), showing the importance of cues regarding motion.
Scaling. In Table 2b, we show the impact of scaling the dataset size and the model size for PlanT and PlanCNN. The circle size indicates the inference time (IT). First, we observe that PlanT demonstrates better data efficiency than PlanCNN, e.g., using the 1× data setting is sufficient to reach the same performance as PlanCNN with 2×. Interestingly, scaling the data from 1× to 3× leads to expert-level performance, showing the effectiveness of scaling. In fact, PlanT MEDIUM (81.36) outperforms the expert (76.91) in some evaluation runs. We visualize one consistent failure mode of the expert that leads to this discrepancy in Fig. 3a. We observe that the expert sometimes stops once it has already entered an intersection if it anticipates a collision, which then leads to collisions or blocked traffic. On the other hand, PlanT learns to wait further outside an intersection before entering which is a smoother function than the discrete rule-based expert and subsequently avoids these infractions. Importantly, in our final setting, PlanT MEDIUM is around 3× as fast as PlanCNN while being 4 points better in terms of the DS and PlanT MINI is 5.3× as fast (IT=5.46 ms) while reaching the same DS as PlanCNN. This shows that PlanT is suitable for systems where fast inference time is a requirement. We report results with multiple training seeds in the supplementary material.

Loss.
A detailed study of the training strategy for PlanT can be found in the supplementary material, where we show that the auxiliary loss proposed in Eq. (4) is crucial to its performance. However, since this is a self-supervised objective, it can be incorporated without additional annotation costs. This is in line with recent findings on training transformers that show the effectiveness of supervising multiple output tokens instead of just a single [CLS] token [73].   (Table 2a) and architecture (Table 2b). for learning-based planners. Including vehicles to the back of the ego, encoding vehicle speeds, and scaling to large models/datasets is crucial for the performance of both PlanCNN and PlanT.

Combining an Off-the-Shelf Perception Module with PlanT
Next, we discuss the results of the sensor-based methods in Table 1. We compare the proposed approach to LAV [30] and TransFuser [26], which are recent state-of-the-art sensor-based methods. Our perception module is based on TransFuser, enabling a fair comparison to this approach. Therefore, PlanT with perception only detects vehicles to its front and has a limited view to the sides (16m instead of 30m

Explainability: Identification of Most Relevant Objects
Finally, we investigate the explainability of PlanT and PlanCNN by analyzing the objects in the scene that are relevant and crucial for the agent's decision. In particular, we measure the relevance of an object in terms of the learned attention for PlanT and by considering the impact that the removal of each object has on the output predictions for PlanCNN. To quantify the ability to reason about the most relevant objects, we propose a novel evaluation scheme together with the Relative Filtered Driving Score (RFDS). For the rule-based expert algorithm, collision avoidance depends on a single vehicle which it identifies as the reason for braking. To measure the RFDS of a learned planner, we run one forward pass of the planner (without executing the actions) to obtain a scalar relevance score for each vehicle in the scene. We then execute the expert algorithm while restricting its observations to the (single) vehicle with the highest relevance score. The RFDS is defined as the relative DS of this restricted version of the expert compared to the default version which checks for collisions against all vehicles. We describe the extraction of the relevance score for PlanT and PlanCNN in the following. Our protocol leads to a fair comparison of different agents as the RFDS does not depend on the ability to drive itself but only on the obtained ranking of object relevance.  Baselines. As a naïve baseline, we consider the inverse distance to the ego vehicle as a vehicle's relevance score, such that the expert only sees the closest vehicle. For PlanT, we extract the relevance score by adding the attention weights of all layers and heads for the [CLS] token. This only requires a single forward pass of PlanT. Since PlanCNN does not use attention, we choose a masking method to find the most salient region in the image, using the same principle as [59,58,57]. We remove one object at a time from the input image and compute the L 1 distance to the predicted waypoints for the full image. The objects are then ranked based on how much their absence affects the L 1 distance.
(a) PlanT vs. Expert. PlanT waits further outside the intersection than the expert to avoid a collision.
(b) RFDS. Vehicles with the highest relevance score are marked with a red bounding rectangle. We show examples for successful matching of relevance score and intuition (green frames) and failures (red frames). Figure 3: We contrast a failure case of the expert to PlanT (Fig. 3a) and show the quality of the relevance scores (Fig. 3b). The ego vehicle is marked with a yellow triangle and vehicles that either lead to collisions or are intuitively the most relevant in the scene are marked with a blue box.
Results. We provide results for the reasoning about relevant objects in Table 3. Both planners significantly outperform the distance-based baseline, with PlanT obtaining a mean RFDS of 96.82

Conclusion
In this work, we take a step towards efficient, high-performance, explainable planning for autonomous driving with a novel object-level representation and transformer-based architecture called PlanT. Our experiments highlight the importance of correctly encoding the ego vehicle's route for planning. We show that incorporating a 360°field of view, information about vehicle speeds, and scaling up both the architecture and dataset size of a learned planner are essential to achieve stateof-the-art results. Additionally, PlanT significantly outperforms state-of-the-art end-to-end sensorbased models even with a noisy and incomplete input representation obtained via a perception module. Finally, we demonstrate that PlanT can reliably identify the most relevant object in the scene via a new metric and evaluation protocol that measure explainability.
Limitations. Firstly, the expert driver used in our IL-based training strategy does not achieve a perfect score (Table 1) and has certain consistent failure modes (Fig. 3a, more examples in the supplementary material). Human data collection to address this would be time-consuming (the 3× dataset used in our experiments contains around 95 hours of driving). Second, all our experiments are conducted in simulation. Real-world scenarios are more diverse and challenging. However, CARLA is a high-fidelity simulator actively used by many researchers for autonomous driving, and previous findings demonstrate that systems developed in simulators like CARLA can be transferred to the real world [31,74,75]. Finally, our experiment with perception (Section 4.1) uses a single offthe-shelf perception module that was not specifically optimized for PlanT, leading to sub-optimal performance. It is a well-known limitation of modular systems that downstream modules cannot recover easily from errors made by earlier modules. A thorough analysis of perception robustness and uncertainty encoding are important research directions beyond the scope of this work.