{'fraction_non_alphanumeric': 0.030596717510235138, 'fraction_numerical': 0.013966885257780516, 'mean_word_length': 4.822487079422002, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Text2Pos: Text-to-Point-Cloud Cross-Modal Localization

Natural language-based communication with mobile devices and home appliances is becoming increasingly popular and has the potential to become natural for communicating with mobile robots in the future. Towards this goal, we investigate cross-modal text-to-point-cloud localization that will allow us to specify, for example, a vehicle pick-up or goods delivery location. In particular, we propose Text2Pos, a cross-modal localization module that learns to align textual descriptions with localization cues in a coarse- to-fine manner. Given a point cloud of the environment, Text2Pos locates a position that is specified via a natural language-based description of the immediate surroundings. To train Text2Pos and study its performance, we construct KITTI360Pose, the first dataset for this task based on the recently introduced KITTI360 dataset. Our experiments show that we can localize 65% of textual queries within 15m distance to query locations for top-10 retrieved locations. This is a starting point that we hope will spark future developments towards language-based navigation.

. We propose Text2Pos for city-scale position localization based on textual descriptions. Given a point cloud that represents our surroundings and a query position description, Text2Pos provides the most-likely estimate of the described position within that map.

Abstract
Natural language-based communication with mobile devices and home appliances is becoming increasingly popular and has the potential to become natural for communicating with mobile robots in the future. Towards this goal, we investigate cross-modal text-to-point-cloud localization that will allow us to specify, for example, a vehicle pickup or goods delivery location. In particular, we propose Text2Pos, a cross-modal localization module that learns to align textual descriptions with localization cues in a coarseto-fine manner. Given a point cloud of the environment, Text2Pos locates a position that is specified via a natural language-based description of the immediate surroundings. To train Text2Pos and study its performance, we construct KITTI360Pose, the first dataset for this task based on the recently introduced KITTI360 dataset. Our experiments show that we can localize 65% of textual queries within 15m distance to query locations for top-10 retrieved locations. This is a starting point that we hope will spark future developments towards language-based navigation.

Introduction
Future mobile robots, such as autonomous vehicles and delivery drones, will need to cooperate with humans to coordinate actions and plan their trajectories. In this paper we tackle large scale position localization of the target position based on natural-language-based position descriptions, as needed for, e.g., for goods delivery or for vehicle pickup.
For self-localization within a map, mobile agents rely on visual localization methods [4,22,36,42,57]. These methods match observed images either to a database of geotagged images [4,16,46] or point-cloud-based maps [34,36,40], often obtained using structure-from-motion techniques [39,41]. By contrast, in this paper, we study language-based localization of any location, which, importantly, does not require the user to be physically present at the target location. This would, for example, allow us to explain the pick-up position or delivery location through text/voice to a robo-taxi via natural language based communication, that is preferable to humans. Our method can also be seen as complementary to GPS localization methods, e.g., when a GPS tag is too coarse, unavailable, or language-based communication is more convenient.
As the main contribution of this paper, we formalize the task of language-based localization and provide the first dataset and methods for this task. In this problem setting, we assume an intelligent agent is given access to the map of the environment that comes in the form of a 3D point cloud and object instance labels. While there are several ways of acquiring point clouds, we rely on LiDAR point clouds, readily available in modern automotive [8,44,53] and urban [27] datasets, On the query side, we assume a textual description of the query position surroundings, such as the one shown in Fig. 1. The task is then to provide the most likely position estimate based on this query.
To study this challenging problem, we need a dataset that (i) provides a point-cloud-based representation of the environment and (ii) provides labels in the form of query positions and their corresponding textual descriptions, extracted from their immediate surroundings. We build on the recently proposed KITTI360 dataset [53], which provides nine scenes (city districts), covering 80 km of driving data. Importantly, this dataset provides semantic and instance-level annotations of the point cloud, which we use to automate the generation of query position descriptions. We obtain the KITTI360Pose dataset by randomly sampling query positions and by generating corresponding textual descriptions. For each query position we automatically generate multiple descriptions based on a natural language template that specifies spatial relations of surrounding instances to the query position, together with their semantic classes and appearance. We generate 43, 381 such descriptions for 14, 934 sampled positions, which we split by scene (representing a city district) to obtain our train/test splits.
We use this dataset to train and evaluate our proposed Text2Pos model that performs coarse-to-fine localization. In the coarse localization step, we retrieve sub-regions of the map that likely contain our target position. To this end, our network learns to align the encoded query with the point clouds, representing these sub-regions. We finally refine the position estimate within retrieved candidate regions using our matching-based fine localization module. Our experiments show that we can localize such randomly generated positions within KITTI360 scenes with 65% recall for top-10 queries, demonstrating that localizing positions based on textual descriptions is feasible.
In summary, our main contributions are: we (i) introduce and formalize the task of 3D point-cloud based localization based on textual descriptions. To this end, we (ii) provide KITTI360Pose, the first public dataset for this task, based on the KITTI360 dataset, together with our method for automated mining of positions and corresponding textual descriptions. We (iii) provide a coarse-to-fine baseline model for the localization task, that learns to align objects which are mentioned in the text with object instances in the point cloud and thoroughly evaluate and ablate the performance on this challenging new task. We believe this work is the first step towards natural language-based communication with future mobile robots, such as delivery drones and self-driving taxis.

Related work
Vision-based localization. Related to our problem is the task of visual localization [4,6,16,22,33,34,36,42,46,47,57], which means estimating a precise pose based on an observed image or image sequence. Existing methods commonly adopt a two-stage coarse-to-fine localization pipeline [34,36,57]. Given the query image, a coarse step firstly finds a subset of images with aligning views using image retrieval techniques [4,16,46]. Then, the fine step establishes 2D-2D correspondences between pixels of the query and the retrieved images based on the visual descriptors. These can further be used to obtain 2D-3D correspondences between the query and a 3D map, usually obtained using structure-from-motion techniques. Finally, the camera poses can be computed using either a set of 2D-2D correspondences [57] or 2D-3D correspondences [34,36]. Our method follows the coarse-to-fine localization scheme by first localizing a coarse cell, containing the objects described by the query text, followed by a more precise pose estimate within the coarse cell. Compared to matching between visual features, our method needs to implicitly learn to align two different modalities: text and 3D point clouds. By contrast to visual localization, commonly used for robot self-localization, we tackle linguistic localization, intended to specify an arbitrary target location.
2D vision and language. Vision and language understanding has been widely investigated in tasks such as image captioning [20,25,48,54], visual question answering (VQA) [3,51] and visual grounding [18,21,26,55], the task of localizing visual elements in the images that are linguistically described by the query text. Visual and linguistic perceptions are combined to assist the task of robot navigation from room to room under building-scale environments [2]. The ALFRED benchmark [43] was later released to encourage research on connecting language to a series of human daily tasks in an interactive visual 3D environment. Closer to our task is text-to-image retrieval [19,28,29,49,50], where text descriptors are learned to match corresponding image descriptors. This usually requires the model to reason about the relationship between a set of words and image regions and match a word to its corresponding image region [29]. The main difference between our approach and previous work is that we match from text to point clouds instead of images. In the coarse localization stage, our method first matches a sequence of texts to a cell which represents a region in the scene and contains a set of objects that are then matched to individual textual object hints.
3D vision and language. Motivated by the 3D world we live in, recent work explores the potential of 3D vision and language understanding on the tasks of 3D shape generation [11] and language grounding of 3D objects [1,10,15,31,56]. The method by [31] embodies language grounding implicitly on 3D visual features and predicts 3D bounding boxes for target objects of primitive shapes of different colors. ScanRefer [10] localizes 3D objects referred by the query descriptions in real-life indoor scenes. ReferIt3D [1] tackles a similar task, but assumes to be given segmented object instances in a room and focuses on identifying the referred object among instances of the same finegrained category. InstanceRefer [56] improves their performance by using a 3D panoptic segmentation backbone, guiding the model to capture multi-level visual context. Recent work by [15] proposes several graph modules that aid learning of the contextual information for both visual and language domains. Similar to our work, these methods localize regions in 3D point clouds based on textual queries. However, this is different to our proposed city-scale text-based position localization task in several aspects: For 3D object reference, a model needs to learn to align a natural-language descriptor to one of the objects in the scene. Different to that, our model needs to learn to interpret a composition of objects as a location and distinguish it from other possible locations, since there is no explicit visual notion of a position. Additional challenges stem from the fact that we are targeting large (city) scale outdoor scene localization. This is challenging due to memory constraints of modern GPUs, which can fit only a very small portion of a city scale point cloud. Furthermore, outdoor regions are less diverse in terms of semantics compared to cluttered man-made indoor environments [30], making it difficult to leverage semantic instances to obtain a unique position signature. In summary, our work serves as the first attempt to tackle this challenging task and opens the door to natural language based localization for the 3D vision and language community.

The KITTI360Pose dataset
To tackle language-based position localization in largescale environments such as urban cities, we (i) need a largescale dataset that provides point clouds representing realworld cities and (ii) a large set of position-text pairs to train and evaluate our models. To this date, no such dataset exists, and hand-annotating textual queries would be prohibitively expensive.
The recently introduced KITTI360 [53] 1 dataset provides nine static scenes that represent different districts of the city of Karlsruhe, covering in total over 80km of driving distance. These scenes were obtained by registering LiDAR scene scans using LiDAR SLAM methods (e.g., [5]). These point clouds would be suitable for studying this problem; however, the dataset does not provide textual descriptions of positions. Luckily, KITTI360 provides object instance labels for static (e.g., building, traffic light, garage) and dynamic (e.g., person, car, bicycle) object instances and semantic labels for the stuff classes (e.g., road, vegetation, wall). In the following, we utilize these object instance and semantic labels to automatize the generation of positiondescription query pairs. We use these to train our models and to benchmark large-scale cross-modal localization without manual annotation work.
In this study, we focus on point clouds, recorded by Li-DAR sensors, readily available in modern automotive [8,44,53] and robotics [27] datasets. Our approach would also be applicable to point clouds obtained using structurefrom-motion methods [39,41] available in existing visual localization datasets [22,37]. However, such datasets currently do not contain appropriate instance annotations that we could utilize to automatize the generation of query positions. Finally, we note that indoor RGB-D datasets, such as [9,12] do contain such object instance labels. However, in this paper, we explicitly aim to study large-scale localization, hence, focus on outdoor scenarios.

Dataset Generation
Object instances. Contrary to the majority of existing automotive datasets [8,44], that focus on instance segmentation of dynamic objects such as cars and pedestrians, KITTI360 additionally provides object instance labels for several static classes, such as buildings and traffic lights, that provide a reliable cue for localization. In this work, we leverage static object instances to generate position queries and as cues for the position localization.
In addition to labeled instances, we also further split certain stuff classes and use the obtained clusters to generate descriptions. For example, the class vegetation aggregates a large set of separate trees and bushes which could be specified as localization cues into a single object that spreads across the entire scene. In order to instead recover a set of separate and therefore localizable instances, we cluster all stuff classes, such as vegetation, fence and wall using the DBSCAN [14] algorithm. We provide further details on the clustering procedure in the supplementary material.
Query generation. The next step is position-query pair generation. The aim here is to obtain a set of positions and corresponding texts that describe each position qualitatively based on the surrounding objects and their spatial relations in an automated fashion. We start by sampling equidistant locations along recorded vehicle trajectories, readily available with maps. In the vicinity of each sampled location, we sample a fixed number of random locations (in practice, 4 or 8) to increase the number of positions. We describe adjacent objects to each position based on their relative position, color and semantic class by generating textual descriptions based on a simple sentence template, describing the position ...

Database of Cells
Top-k Cells Figure 2. Text2Pos. Coarse localization. Given a template-based query position description, we first identify a set of coarse candidate locations (i.e., cells) that potentially contain the target position, which serves as the coarse localization of the query. This is achieved by retrieving top-k nearest cells from our constructed database of cells using our text-to-cell retrieval model. Fine localization. we then refine the pose within retrieved cells via our position refinement module.
in relation to adjacent objects. We extract the relative position and object color tag in an automated fashion directly from the point cloud. We detail the sentence generation in the supplementary.
In the following, we refer to one such generated sentence as a hint. A position description T is defined by a set of hints {h i } describing a given position, i.e., T : where N h is the number of hints per position description. The set of objects to describe a position with is obtained by selecting a set of k objects close to the sampled query position. We retain only positions with at least N h objects in their vicinity. The reason for such well-structured queries is two-fold: (i) this allows us to investigate the problem without costly human annotations, and (ii) enables us to rigorously study cross-modal localization in a well-controlled setting based on explicit hint-to-object matching. We are confident that more complex language queries can be encoded and understood by building on recent developments in the field of natural language processing [7,13].
KITTI360Pose dataset. Using the procedure described above, we generate the KITTI360Pose dataset. In particular, we sample 14, 934 positions and generate up to three descriptions for each, totaling in 43, 381 position-query pairs. We use five scenes (districts) for training (covering in total 11.59km 2 ), one for model validation, and three for testing (covering in total 2.14km 2 ). An average district covers an area of 1.78km 2 . In contrast, the Cambridge dataset [22] covers an area of 0.063km 2 , Oxford RobotCar and CMU Seasons [37] cover 10km and 8.5km of driving distance (respectively), and Tokyo 24/7 [46] covers the area of 2.56km 2 . Descriptions of these positions are generated based on objects that fall within a 15m radius for a sampled position. We provide additional details in the supplementary.

Tex2Pos: A Baseline for Language-based Localization
Given a textual position query, our goal is to localize an agent within a given point-cloud-based (Sec. 3.1) map via 2D planar coordinates of its position w.r.t. the scene coordinate system. To this end, we propose the first text-based coarse-to-fine localization method, that we outline in Fig. 2. Due to the large-scale nature of the problem, we follow a coarse-to-fine framework, well studied and proven successful in the field of large-scale visual localization [34,57]. We first perform a coarse localization of the query where we discretize the search region into rectangular cells and retrieve the top-k cells matched to the description from the database (Fig. 2, left). To refine this estimate, we match the visible 3D instances within a retrieved cell, to their corresponding referring hints in the textual description ( Fig. 2, right). Finally, we obtain the position estimate from the set of instances identified by the text (Sec. 4.2). We note that the coarse retrieval could in the future be replaced by hints such as street name or coarse address. This would require establishing an additional alignment between the 3D point cloud and city map, and remains our future work.

Coarse Localization
Image retrieval techniques [4,16,46] are commonly used within visual localization pipelines to efficiently narrow down the search space [34,45,57] or even provide a direct coarse position estimate [38] of the query image [37]. Given a query image, its learned global descriptor is matched against the global descriptors extracted from a database of reference images to obtain its top-k nearest reference images based on their descriptor distances. We follow this general approach in our text-to-cell cross-modal retrieval method. With this step, we aim to efficiently locate candidate regions within the map that could potentially contain our target position.
Database construction. As a pre-processing step, we divide point clouds, representing city districts, into rectangular cells. We sample cells by sliding a W × W window with a stride of S horizontally and vertically over the scene, where the cell size W should be large enough to contain a certain number of instances, that can be used to describe a position. The stride size S is picked to be smaller than the Instance Descs.  cell size to cover the whole scene area and to allow partially overlapping cells. We consider an instance p i to be inside a cell C if at least a third of its points lie within the cell, or if a minimal number of points (250 in practice) overlap with the cell -this criterion is important for stuff classes, such as tree or building. We name those instances in-cell instances of that cell, i.e., C :

Text-to-Cell
where N p is the number of in-cell instances per cell and varies for different cells.
Text-to-cell retrieval. Given a position description, the task of our retrieval model is to identify its top-k candidate cells that are likely to contain the described position location. Compared to image-to-image retrieval, the model needs to learn to extract descriptors for inputs from two different modalities, i.e., text and point clouds, such that the two can be directly compared using Euclidean distance in the embedding space.
As shown in Fig. 3 (top), our retrieval network has two encoding branches to process a query position description T and a candidate cell C. The complete position description T is encoded into a global text descriptor F T using a bidirectional LSTM cell [17]. On the cell encoding side, we first extract a descriptor F pi for each in-cell instance p i ∈ C. We aggregate in-cell instance descriptors {F pi } Np i=1 into a global cell descriptor F C using an EdgeConv layer [52] followed by a max pooling operation.
Instance encoder. Each instance P i is represented by a point cloud where each point contains three spatial and three color (RGB) coordinates, yielding 6D input features (Fig. 3 bottom). We encode such a point cloud using a PointNet++ [32] backbone, which gives us a semantic embedding. In addition, we explicitly obtain a color embedding of it by encoding its RGB coordinates using our color encoder and a positional embedding of it by encoding its instance centerP i , i.e., the mean value of its coordinates, using our positional encoder. Each of the color encoder and positional encoder takes form of a 3-layer multi-layer perceptron (MLP), whose output dimension is the same as the semantic embedding dimension. The semantic, color and positional embeddings are fused by concatenation and fed  Figure 4. Fine localization. In the fine localization, for each candidate cell, we first establish the correspondences between the query hints and its in-cell instances, which allows us to filter out noisy signals that are not useful for position estimation from both domains. We then predict a vector for each matched instance that translates its instance center to a position estimate via a translation regressor. The final position is the average of the position estimates by all matched instances.
into a projection layer (another 3-layer MLP) which outputs a final instance embedding F pi ∈ R Dp .

Fine Localization
Given a set of retrieved candidate cells, we now find cues that allow us to refine the position among those cells based on the query description. In classical visual localization, such a fine localization step normally relies on establishing 2D pixel correspondences between the query image and retrieved top-k database images [34,57]. These images have geo-information attached and provide cues to compute an accurate camera position for the query. Inspired by this idea, we propose to compute the refined position by establishing hint-to-instance correspondences between a position description and its top-k retrieved cells.
Hint-to-instance matching. Given a position description T and a candidate cell C, we first establish hint-to-instance correspondences with our cross-modal matching module (Fig. 4). As explained in Sec. 3, a position description consists of a series of hints {h j } N h j=1 , where each hint describes a single instance about its attributes and relation to the position. We use our instance encoder to extract a descriptor F pi for each in-cell instance p i ∈ C and our text encoder to extract a descriptor F hj per hint.
We then use a matching module (inspired by Super-Glue [35]) to perform a partial matching between the set of instances and the set of sentences. The matching module first uses several blocks of self-attention layers and crossattention layers to propagate contextual information. The pair-wise similarity scores are computed from the two sets of aggregated descriptors as the cost matrix to an optimal transport (OT) matching layer. We adopt the same OT setup as in [35]. Finally, we obtain the partial assignment from hints to instances by picking matches with confidence scores above the certain threshold (0.2 in practice). (p i , h j ) of the instance p i and the hint h j , we additionally learn a translation vector t i that transforms the instance centerp i to the target position. Each translation prediction t i then leads to a position estimate that is defined as To predict such a translation vector, we train an additional 3-layer MLP translation regressor that takes a hint descriptor F hj as the input, and outputs a translation vector t i . The final position of the query text is computed by taking the average of the set of estimates {ỹ i } predicted from all hint-to-instance matches. We show in our ablation (Sec. 5.1) that our learned translation leads to a more accurate position prediction compared to simple averaging of the matched instance positions.

Losses
Coarse loss. Given an input batch of cell descriptors where N b is the batch size, we train the network for cross-domain retrieval with the pairwise ranking loss [24]: where [•] + equals max(0, •) and α is the margin hyper parameter. This loss enforces that each cell descriptor F i C is closer to its matching text descriptor F i T than it is to nonmatching text descriptors F j T in the batch by a margin. The same is also enforced when matching from a text descriptor F i T to a cell descriptor. Fine loss. To train our matching module, we adopt the matching loss used in [35], which maximizes the scores at ground truth (GT) match locations in the predicted assignment matrix. We train this model separately from the coarse module. To train the translation regressor, we minimize the mean square error between the predicted and the GT translation. Matching module and the regressor are jointly trained with the sum of the matching loss and regression loss. We provide implementation details of model training in our supplementary material.  Table 2. Ablation on the number of hints in a query description.

Experimental Evaluation
In this section, we discuss the performance of our model (Sec. 4) on the proposed KITTI360Pose dataset. We report results on our validation split for ablation studies (Sec. 5.1) and on our test split for a final evaluation of our best performing models (Sec. 5.2). For details on the dataset and splits, we refer to our supplementary.
Evaluation metrics. We perform the evaluation w.r.t.the top k retrieved candidates (k ∈ {1, 5, 10}) and report localization recall, i.e., the ratio of successfully localized queries if its error is below specific error thresholds, i.e., < 5/10/15m by default.

Model Ablations
Database construction. The coarse localization module (as explained in Sec. 4.1) retrieves the top-k candidate cells.
To study its performance w.r.t. the localization task, we use the center of a cell as a coarse position estimate and measure its accuracy. As our first ablation, we study the impact of the cell sampling stride S on localization performance.
As shown in Tab. 1, the retrieval performance on cells sampled with S = 10m performs better than other stride settings across different k values. As the smaller stride implies more overlap between consecutively sampled cells, it helps our model to learn more discriminative descriptors that can be used to distinguish the content of close-by cells. While decreasing the stride allows for more accurate localization, it increases the computation demand in terms of memory and runtime in quadratic order. We present results for denser sampling using S = 1/3/5m in the supplementary. Considering the trade-off between accuracy and computational efficiency, we use S = 10m to train and evaluate our models in the following experiments.
Ablation on number of localization cues. To evaluate how the number of specified object cues influences localization recall, we vary the number of hints N h from four to 12 either during training or inference, as shown in Tab Table 3. Ablation on fine localization components. This ablation study on the fine localization module requires narrower localization thresholds compared to Tab. 1 to reveal differences in localization precision.
N h = 6 hints is robust to varying N h during inference and that performance even rises by up to 16 points for N h = 12, which is to be expected as additional hints help to resolve some of the ambiguity in cross-modal localization.
Fine localization module. In this ablation, we provide insights into design decisions for the fine localization module (Sec. 4.2). To make the evaluation independent of the retrieval module performance, we use a retrieval oracle in the following experiments to provide the model with the database cell that is closest to the GT position. Firstly, we study the performance of the hints-toinstance matching module, by measuring precision and recall of the predicted matches given the GT matches. Our matching model can achieve 78% and 76% for both recall and precision, respectively.
Next, we show the benefit of our translation regressor by comparing the following three different variants for position estimation: (i) we take the cell-center as the estimated position (no refinements); (ii) we compute the mean of the instances matched by query hints, and (iii) we use the mean of the position estimates computed by using the predicted translation vectors. Due to the use of a retrieval oracle, we report their localization recall with stricter error thresholds < 2/5/10m. As shown in Tab. 3 (top), all three variants can localize queries within 10m errors with almost 100% recall. The difference between model performance becomes more evident when using the smaller error threshold = 2m, where we see the simple cell-center baseline performs poorly to localize only 14% of queries. This only improves marginally by 1% if we take the mean center of matched instances. Additionally, we even see the opposite results for a moderate error of = 5m where the cell-center baseline outperforms the instance-mean baseline by 15%. The results thus suggest that naively assuming either the position to be at the center of the cell or between the described instances is not sufficient. Instead of making such manual assumptions, we learn where to expect the position given an instance of a specific object class in a data-driven manner. We show that our learned translation leads to 24% localization recall on the smallest threshold, outperforming the other two variants by up to 10% and confirming its potential efficacy for fine-grained localization.
Finally, we replace parts of the fine model by oracles to understand the limitations of individual components of our method. As shown in Tab. 3 (bottom), using GT associations (matching oracle) instead of performing hint-toinstance matching using our model, we localize (10%/15%) more queries within 2/5m errors. Purely replacing predicted translation with GT translations (translation oracle) leads to 31% more queries localized within 2m, suggesting room for improvements by predicting more accurate translation vectors. Both oracles combined yield perfect localization as expected.
Pipeline-level oracle ablation. While the previous experiments used a coarse oracle to evaluate fine localization in isolation, we now replace first our coarse and then our fine module by oracles, in order to understand their limitations of the full system. The results are shown in Tab. 4. With coarse oracle, we replace the retrieval component with a retrieval oracle (as in the previous fine module ablation). As can be seen, identifying cells reliably localizes all queries within 15m distance to the query position, and with 98% accuracy within 10m threshold. This clearly shows significant potential in improving the retrieval module, which can be achieved potentially with additional information (such as nearby landmarks or street names) to reduce the inherent cell ambiguity. This remains our future work. With fine oracle, we then measure the localization performance of our learned retrieval model + GT matching association + GT translation prediction. We show that perfect fine localization improves the baseline by 11%/29%/31% at the smallest thresholds for k = 1/5/10 candidate cells, which indicates further potential in improving the refinement module. Finally, using both oracles, we can localize all queries within 5m distance from the query position. For the lower part of Tab. 4, we also replace our coarse and fine modules with random choice to highlight their importance. Compared to our baseline, a random coarse retrieval leads to near-zero localization recall, while a random matching in the fine module leads to localizing up to 34 % less queries.
Comparison to visual localization While Text2Pos is not intended as a competitor against visual localization, we do provide a comparison in localization recall through a twostep experiment (Tab. 5). Visual localization has to be performed on rendered images in order to be applicable on our dataset, so we first show a comparison between real and rendered images from identical locations, confirming that rendered images do not significantly reduce retrieval performance. Then, we compare visual retrieval (NetVLAD [4] 2 ) to our cross-modal retrieval by rendering images at each cell and query pose of our validation set, resulting in 5736 and 3187 images on the database and query side, respectively.
The results indicate that visual retrieval shows superior accuracy for smaller top-k values with up to 13 percentage points at k = 1 and = 15m, but also lags behind crossmodal retrieval with up to 6 percentage points in k = 10 and = 10m. We note that for simplicity, this comparison considers retrieval only without refinement and that more advanced visual localization pipelines are available. Nevertheless, we take this as a first indication that equally strong advancements in cross-modal localization might approach the accuracy of state-of-the-art visual pipelines.

Evaluation on the Test Set
As can be seen in Tab. 6, for top-1 retrieval, we can successfully localize up to 25% of queries up to 15m distance threshold and 20% of queries up to a distance of 10m. When considering top-5 queries, we can already successfully localize 52% of all queries and go up to 65% successfully localized queries when considering top-10 cell candidates. As can be seen, based on purely textual descriptions, we can already localize the target position quite accurately (i.e., 10m radius). However, to get to a good localization recall, we need to consider top-10 cell candidates. This suggests that our model is rather uncertain which cell contains the query position. This is not surprising, as many of the query descriptions could refer to several locations in the city. Additional hints such as a nearby address, nearby street names, or landmarks should help in the future to further narrow down the search space and resolve this ambiguity and provide a good direction for future research.
Finally, as can be seen in Tab. 6, the full localization variant using translation prediction outperforms the variant using naive matched-instances position averaging by localizing 3% more queries within 5m errors considering top-1 retrieved cell. This improvement increases to 8% when considering top-10 candidates.

Broader Impact and Limitations
Our work on text-based localization opens a new front of research on natural-language-based action coordination with future mobile systems, to which we may need to specify our current or a target location. Usage scenarios include autonomous goods delivery (e.g., food or packages), ordering autonomous vehicle pick-up, or sending vehicles to remote locations in case of emergency. These use-cases will play an important role in the automation of tasks that can be considered unsafe (due to a high number of traffic accidents) and currently require a human operator. Automation of these tasks can also improve the weight and utilization of delivery vehicles and, consequentially, the carbon footprint.
However, to devise a first feasible solution, we rely on the following assumptions: (i) we assume our maps contain labeled instances of objects used as anchor points for localization; (ii) we rely on simplistic, template-based localization instructions for training and evaluation. Generalization to arbitrary, unlabeled point clouds and more realistic, human-generated textual queries remains our future work.

Conclusion
We presented Text2Pos and KITTI360Pose, the first method and dataset for text-based position localization within a 3D environment. As such language-based communication is natural to human beings, we foresee it will be an integral part of future mobile agents that will require location instructions, such as goods delivery or vehicle pickup positions. We demonstrated that our coarse-to-fine approach can localize 65% of textual queries within 15m distance to query locations when considering top-10 retrieved locations. We believe we can further improve localization precision by using street names and visual landmarks as cues for coarse localization, which we leave for future work. This work is the first step in the direction of language-based localization, showing its great potential, and hopefully inspiring researchers to further make this technology a reality.
In this supplementary material, we provide additional information to further understand our proposed coarse-to-fine language-based localization model -Text2Pos. In Sec. A, we provide more details about our KITTI360Pose dataset such as the query position sampling mechanism and the clustering and describing of scene object instances. Next, we describe implementation details about the cell database construction, data augmentation, model training and PointNet++ pre-training in Sec. B. We further present thorough ablation experiments to study the impact of data variations on the localization performance in Sec. C. There we also prove our concept of utilizing street names (when available) to better overcome the ambiguity in descriptions caused by the fact that many locations have semantically similar or equal surroundings. Finally, Sec. D shows qualitative results of top-k candidate cells retrieved during the coarse localization stage. We will release our code upon the paper's acceptance.

A. KITTI360Pose Dataset Details
Query position sampling and description. To obtain query positions, we first sample equidistant locations along the original KITTI360 [53] capturing paths. For each of these trajectory locations, we then sample a number of random nearby positions that are up to 1 /2 cell-sizes away from it. We sample 4 positions for the baseline and present detailed studies on the impact of training our baseline on a larger dataset that samples more query positions (per trajectory location) in Tab. 9.
As detailed in the main document, a query position is described based on the set of its surrounding object instances, and thus we discard a sampled query position if there are insufficient instances, i.e. N h < 6 in practice, in its vicinity. Given a query position with enough nearby instances, we can select the subset of the instances (6 instances in practice) to describe the query using three different strategies: (i) simply selecting the closest-by instances, (ii) choosing the instances that cover as many different directions (pointing towards the query position) as possible and (iii) picking instances with as many different classes as possible. If two or more of these strategies yield the same set of instances, the duplicates are omitted.
Object clustering. In order to also incorporate stuff class objects into our descriptions, it is necessary to cluster them into separate instances, e.g., clustering one large sidewalk object covering the entire scene into separate instances left and right of a query position. Since we did not find a way to achieve such a clustering globally for a whole scene, we decide to cluster a stuff-class object locally for a given cell, where we first ignore those points of the complete stuff object that are not contained in that cell. For its points inside the cell, we then cluster them into multiple separate instances using the DBSCAN [14] and retain the clustered instances that contain enough points (250 in practice). Dataset Statistics. As described above, we can obtain datasets of different scales by varying the number of sampled positions at each trajectory location. In Tab. 7, we show statistics about the number of query positions, descriptions and covered scene area of the two Kitti360Pose dataset versions generated by sampling 4 and 8 positions per trajectory location.
Challenges. Notice, the mapping between a query description and a target position is often not unique, since there exist many positions with semantically similar or even equal surroundings. Such ambiguous nature makes the task of text-based outdoor localization very challenging. Compared to the indoor environment which is rich in semantics of various objects such as table, sofa, chair and cup [12], outdoor scenes have fewer static objects, yet more complicated semantic cues that are mainly based on stuff classes, such as tree, vegetation, fence and road. Those stuff -based semantic features are highly repetitive for real-world large-scale outdoor scenes, which is also verified by our statistics in Tab. 7, which show that only a small fraction of the generated descriptions is actually unique. Therefore, our method is designed to endure such ambiguity by retrieving several candidate cells and then performing more refined pose estimations for each of them. In future work, we expect to reduce the ambiguity by incorporating the use of unique landmarks like street names or named buildings, which can be easily integrated into our general coarse-to-fine localization pipeline. We present our experiment in Tab. 10 as a proof of concept, where we manage to utilize simulated street names to improve our localization performance up to 20 percent points.

B. Implementation Details
Cell database. Our method relies on a database of cells to first retrieve top-k candidate cells which potentially contain our target position and then perform more accurate fine localization within those cells. To construct a database of cells that can fully cover the scene area, we use a sliding window with size W and sliding stride S to sample the cells along both the horizontal and vertical directions. We empirically fix the cell size to be W = 30m which usually covers enough instances for our experiments and we use a stride of S = 1 /3 × W. After the raw sampling, we further reject cells that have not enough objects inside to describe a position, e.g. a cell with mostly empty space. This leads to a database of 11259/1434/4308 cells for the training/validation/testing scene split and in total 17, 001 cells for the whole dataset.
In-cell instances. For fine localization, we cut-off or pad in-cell instances to keep the same number of instances per cell to allow mini-batching in the matching module. We consider each cell to contain 16 in-cell instances, i.e., Np = 16 for both training and inference, which has led to the most promising performance across different data settings in our experiments. In the case of too few instances, we pad dummy instances of 10 points which have black color and random point coordinates close to zero so that they are easily recognized by the model. In addition, we normalize the coordinates of in-cell instances w.r.t. the size of their belonging cell such that each coordinate value is ∈ [0, 1], which aids the regressor's training stability.
Query description grounding. To learn the task of text-to-cell retrieval, we need to ground a query position description onto a ground-truth (GT) cell. We define a GT cell to be the cell in the database that contains the described position and whose center is closest to the described position. To learn the task of hint-to-instance matching, we need to further identify the GT correspondences between the query hints and the in-cell instances within a candidate cell. For instance-class instances, the hint-to-instance correspondences are established using GT instance IDs from the original KITTI360 dataset. Notice, stuff -class hint instances were clustered using a synthetic cell centered on the query position, which means the same instance might appear in a slightly different position within a retrieved cell since the query position can be anywhere inside that retrieval cell. Therefore, to match between a stuff -class hint instance and a point-cloud in-cell instance which are clustered w.r.t. a synthetic cell and a retrieval cell, we compare their semantic class IDs and the two direction vectors pointing from the query position to those two instances. We consider them as a match, if they have the same semantic class and their direction vectors are close enough.
Data augmentation. During training, we also use data augmentations for the point cloud instances and the pose descriptions to aid our model performance. In the instance augmentation, we randomly rotate the instance across the z-axis during training and normalize-scale all its points to a [0, 1] interval during the training and inference. For the description augmentation, we (i) randomly shuffle the order of the hints that make up a query description and (ii) randomly flip cells horizontally and/or vertically by flipping the location of the pose and instance centers in each cell and changing the corresponding words in the description. The description augmentations are not used when training the refinement module.
Text2Pos model training. We train our model using five training scenes and use one scene for model validation. For coarse localization, we train the retrieval model using an Adam optimizer [23] with batch size 64 and learning rate 0.001 for at most 64 epochs. We set the margin parameter α in the pairwise ranking loss (Eq. (1)) as α = 0.35. For fine localization, we train the matching module and regression jointly using an Adam optimizer with batch size 32 and learning rate 0.0003 for 16 epochs.  Table 9. Training on larger dataset. We generate a larger dataset for training by varying the number of positions sampled at every trajectory location.
PointNet++ pretraining. We pre-train our PointNet++ [32] backbone for the point cloud classification on KITTI360 and use it to initialize the two instance encoders used in the coarse and fine localization stages. We aggregate the objects from all our database-side cells into a training set of 159, 828 objects from 22 classes and again use the Adam optimizer [23] with a learning rate of 0.003 and a batch size of 32.

C. More Ablation Studies
In the main document, we have presented several ablation studies that focus on analysing the localization performance of each component of our proposed Text2Pos model, i.e., the coarse retrieval component and the fine matching component. In this section, we first complete the results of the cell stride ablation (that has been presented in Tab. 1 of the main paper). And we further provide an additional ablation to study the impact of training on a larger version of our dataset on the localization performance, to thoroughly understand our proposed method. Finally, we confirm our hypothesis that our localization performance can be further improved when provided with additional landmarks information such as street names.
Full results on sliding stride ablation. We variate the stride size for our database-side cells between values decreasing from S = 20m down to 3m, while keeping a fixed cell size of 30m. As shown in Tab. 8 where we mark the best recall with bold, our baseline setting S = 10m leads to the best performance at 10/15m error thresholds when using top-5/10 candidates. For 5m errors or top-1 candidates, the recall steadily increases for smaller strides, e.g., from 7% up to 22% recall for a top-1 candidate prediction within a 5m error threshold. However, we consistently observe a larger performance drop (if any) when going from S = 15m to 10m than going from S = 10m to 5m. While, S = 3m is the most promising one for the finest error threshold, it leads to less efficient computation in both memory and run time due to the large amount of generated cells (11 times more than in the 10m setting). For our current implementation, the clustered stuff instances are pre-computed and stored separately for each cell, meaning it takes approximately 9 GB of memory to store 15k cells (0.6MB per cell) just for the validation split. Another option to avoid big memory consumption is to perform instance clustering on-the-fly during training and inference, however, loading plus clustering of all cells in the validation split already takes around 50 minutes (0.2s per cell). This also prevents us from pushing the stride to more extreme settings, e.g., S = 1m where one needs to handle approximately 135k cells for the validation split. To maintain the feasibility of performing intensive evaluations, we choose our baseline setting as S = 10m. This gives us the best trade-off between computational efficiency and localization recall, as we consider further technical engineering to improve the computational overhead coming from instance clustering out of the scope of this research, leaving it to future work.
Using a larger dataset. As described before, it is possible to increase our dataset scale by sampling more poses around each trajectory location. In our final experiment, we vary this scale by sampling 8, 12 and 16 poses per location and train our coarse and fine models on these larger datasets. To maintain the comparability, we evaluate models trained under varying settings on the same validation set that is obtained using the baseline configuration. As shown in Tab. 9, localization recall improves incrementally by training on a larger dataset, acquired by sampling poses from 4 to 12 positions per trajectory location. However, the performance starts to decrease on the 16-positions sampling configuration, which indicates a limit of performance improvement by sampling denser query positions. Our intuition Figure 5. Simulated streets. Overview: We split our validation scenes into nine hypothetical streets marked with different colors. Zoom-in: Each colored square placed on top of a street represents a retrieval cell belonging to that street area.
is that the increase in sampling density leads to increases in the ambiguity of the mapping between a query description and a position, which starts to harm the learning of the global mapping on the cell level and the local mapping on the instance level. We note that we chose the sub-optimal setup with 4 poses per location as our experimental baseline due to time constraints, but will publish our results based using a more optimal sampling in our camera ready version.
Using street names as additional cues. As mentioned in the main paper as well as Sec. A, our method can potentially use additional information such as nearby landmarks or street names to reduce the inherent cell ambiguity caused by places with semantically similar surroundings. To confirm this hypothesis we perform an additional experiment in which we use simulated street names as our additional cues to improve our localization recall. To simulate street names, we split our validation scene into nine separate streets as shown in Fig. 5 (each street in a unique color). All database-side cells and query-side poses are then assigned to their corresponding streets. During inference, we still perform coarse retrieval as usual, but additionally reject all cells from incorrect streets before taking the top-k retrievals. Hence, this experiment serves as a "semi-oracle" for the coarse retrieval. As Tab. 10 shows, using the simulated street names (as prior ground truth) boosts the performance of our baselines up to 20 percentage points. With this improvement, our full pipeline is able to localize 19% of the queries with top-1 retrieval within a strict error radius of = 5m. If we consider all top-10 retrievals, it then manages to localize 61/85/90% of the queries at different error thresholds of 5/10/15m. Therefore, our experimental results indicate that additional text-based cues like street names are valuable information to complement the challenge of description ambiguity. We believe our proof of concept shows this direction is worthy of further research investigation to properly incorporate existing landmark information such as street names, zip codes or named buildings, into text-based localization pipelines for performance improvement.

D. Qualitative Text-to-Cell Retrieval Analysis
Finally, we show examples of the top-3 candidate cells retrieved by our text-to-cell retrieval model in Fig. 6. As we can see in both the successful (left) and failure cases (right), top-3 retrieved cells often exhibit high similarity in semantics to the query cell and to each other, yet can be more than 100m apart, which again highlights the inherent ambiguity of largescale outdoor text-based localization. As a consequence, as shown in the 2nd successful example (left), the correct cell is retrieved but ranked lower (as the 3rd) than the other two candidates which are much further away. Furthermore, the 2nd failure example (right)) shows that the top-1 candidate is indeed a close-by candidate yet will not be considered as a correct one according to our manual 10m threshold, meaning the pose error is too coarse. This also suggests the need of another refinement step to improve the localization recall, which will turn such a case into a successful localization. Top-1  Top-2  Top-3  Query  Top-1  Top-2  Top-3  In the right part, we show 3 failure cases where none of the top-3 candidates is the correct one. Each example consists of the dataset cell that is closest to the query pose (the 1st column) and the top-3 retrieval cells, where the in − cell instances are colored by their semantic classes (the top row) and RGB values (the bottom row). We plot the mapping from semantic colors to class labels in the center of the figure. We further mark the correct cells with green borders and wrong cells with red borders, where we consider a cell to be correct if its center is at most 10m away from the query position (the red dot in the query cell).