{'fraction_non_alphanumeric': 0.03739879670207235, 'fraction_numerical': 0.01311000519943549, 'mean_word_length': 4.808239861949957, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval

Cross-view image geo-localization aims to determine the locations of street-view query images by matching with GPS-tagged reference images from aerial view. Recent works have achieved surprisingly high retrieval accuracy on city-scale datasets. However, these results rely on the assumption that there exists a reference image exactly centered at the location of any query image, which is not applicable for practical scenarios. In this paper, we redefine this problem with a more realistic assumption that the query image can be arbitrary in the area of interest and the reference images are captured before the queries emerge. This assumption breaks the one-to-one retrieval setting of existing datasets as the queries and reference images are not perfectly aligned pairs, and there may be multiple reference images covering one query location. To bridge the gap between this realistic setting and existing datasets, we propose a new large-scale benchmark -- VIGOR -- for cross-View Image Geo-localization beyond One-to-one Retrieval. We benchmark existing state-of-the-art methods and propose a novel end-to-end framework to localize the query in a coarse-to-fine manner. Apart from the image-level retrieval accuracy, we also evaluate the localization accuracy in terms of the actual distance (meters) using the raw GPS data. Extensive experiments are conducted under different application scenarios to validate the effectiveness of the proposed method. The results indicate that cross-view geo-localization in this realistic setting is still challenging, fostering new research in this direction. Our dataset and code will be publicly available.


Introduction
The objective of image-based geo-localization is to determine the location of a query image by finding the most similar image in a GPS-tagged reference database. Such technologies have proven useful for accurate localization with noisy GPS signals [4,26] and navigation in crowded cities [12,9]. Recently, there has been a surge of interest in cross-view geo-localization [24,22,7,17,28,21], which uses GPS-tagged aerial-view images as reference for streetview queries. However, the performance may suffer from a large view or appearance gap between query and reference images.
Recent works [7,17,28] have shown that the performance of cross-view image matching can be significantly improved by feature aggregation and sample mining strategies. When the orientation of street-view (or ground-view) image is available (provided by phone-based compass), state-of-the-art methods can achieve a top-1 retrieval accuracy over 80% [17], which shows the possibility of accurate geo-localization in real-world settings. However, existing datasets [24,27,11] simply assume that each query ground-view image has one corresponding reference aerialview image whose center is exactly aligned at the location of the query image. We argue this is not practical for realworld applications, because the query image can occur at arbitrary locations in the area of interest and the reference images should be captured before the queries emerge. In this case, perfectly aligned one-to-one correspondence is not guaranteed.
In light of the novelty of this problem, we propose a new benchmark (VIGOR) to evaluate cross-view geolocalization in a more realistic setting. Briefly, given an area of interest (AOI), the reference aerial images are densely sampled to achieve a seamless coverage of the AOI and the street-view queries are captured at arbitrary locations. In total, 90, 618 aerial images and 238, 696 street panoramas are collected from 4 major cities in the United States (see details in Sec. 3). The new dataset gives rise to two fundamental differences between this work and prior research.
Beyond One-to-one: Previous research mainly focuses on the one-to-one correspondence because existing datasets consider perfectly aligned image pairs as default. However, VIGOR enables us to explore the effect of reference samples that are not centered at the locations of queries but still cover the query area. As a result, there could be multiple reference images partially covering the same query location, breaking the one-to-one correspondence. In our geolocalization method, we design a novel hybrid loss to take advantage of multiple reference images during training.
Beyond Retrieval: Image retrieval can only provide image-level localization. Since the center alignment is not guaranteed in our dataset, after the retrieval, we further employ a within-image calibration to predict the offset of the query location inside the retrieved image. Therefore, the proposed joint-retrieval-and-calibration framework provides a coarse-to-fine localization. The whole pipeline is end-to-end, and the inference is fast as the offset prediction shares the feature descriptors with the retrieval task. Moreover, our dataset is also accompanied with raw GPS data. Thus a more direct performance assessment, i.e. localization accuracy in terms of real-world distance (e.g. meters), can be achieved on our dataset.
Our main contributions can be summarized as follows: • We introduce a new dataset for the problem of cross-view image geo-localization. This dataset, for the first time, allows one to study this problem under a more realistic and practical setting and offers a testbed for bridging the gap between current research and practical applications. • We propose a novel joint-retrieval-and-calibration framework for accurate geo-localization in a coarse-to-fine manner, which has not been explored in the past. • We develop a new hybrid loss to learn from multiple reference images during training, which is demonstrated to be effective in various experimental settings. • We also validate the potential of the proposed cross-view geo-localization framework in a real-world application scenario (assistive navigation) by simulating noisy GPS.

Related Work
Cross-view Datasets. A number of datasets have been proposed for cross-view geo-localization [10,25,24,27,22,11]. Lin et al. [10] consider both satellite images and land cover attributes for cross-view geo-localization. 6, 756 ground-view images and 182,988 aerial images are collected from Charleston, South Carolina. Although the aerial images are densely sampled, they force a one-to-one correspondence between two views and evaluation in terms of distance is not available. The original CVUSA [25] is a massive dataset containing more than 1 million groundlevel and aerial images from multiple cities in the United States. Zhai et al. [27] further make use of the camera's extrinsic parameters to generate aligned pairs by warping the panoramas, resulting in 35,532 image pairs for training and 8,884 image pairs for testing. This version of CVUSA is the most widely used dataset in recent research [7,17,28,14,23] and we refer to it as CVUSA if not specified. Vo [24] consists of about one million image pairs from 11 cities in the United States. The authors randomly collect street-view panoramas and generate several crops from each panorama along with spatially aligned aerial images from Google Maps. Similar to CVUSA, CVACT [11] also consists of aligned panoramas and aerial images with ori-entation information. It has 35, 532 image pairs for training and 92, 802 pairs for testing. In a nutshell, all these datasets consider one-to-one retrieval and none of them provide raw GPS data for localization evaluation in terms of meters. Cross-view Geo-localization. Early works [10,25,24,22] of cross-view geo-localization suffer from low retrieval accuracy mainly because of the significant appearance gap between two views and poor metric learning techniques. With tailored feature extractors and a modified loss function, Hu et al. [7] show the possibility of achieving accurate localization with end-to-end deep neural networks. Several recent methods [14,17] aim to reduce the domain gap by leveraging GANs [6] and polar transformations [18]. Regmi et al. [14] propose to generate the synthetic aerial-view image from the ground-view query with a conditional GAN and adopt feature fusion to achieve better performance. SAFA [17] further takes advantage of the geometric prior knowledge by applying a polar transformation on the query image and replacing the global pooling with feature aggregation blocks. The top-1 accuracy of [17] on CVUSA [27] is almost 90% if the orientation information is given. Other approaches [5,28] exploring metric learning techniques (e.g. hard samples mining strategy) also show promising results on popular datasets, and they are not restricted by the geometric assumptions. However, none of these methods consider a sub-image level localization beyond the image-level retrieval or multiple reference images for training.

VIGOR Dataset
Problem Statement. Given an area of interest (AOI), our objective is to localize an arbitrary street-view query in this area by matching it with aerial reference images. To guarantee that any possible query is covered by at least one reference image, the reference aerial images must provide a seamless coverage of the AOI. As shown in Fig. 1 (a), coarsely sampled reference images (black square boxes) are not able to provide full coverage of the AOI, and an arbitrary query location (the red star) may lie in the area between reference samples. Even if the query location (the yellow star) lies at the edge of a reference aerial image, this reference image only shares partial (at most half) scene with the one whose center is at the query location, which may not provide enough information to be distinguished from other negative reference images. These queries can be covered by adding additional overlapping samples (the green box). As shown in Fig. 1 (b), if query locations (red stars) lie at the central area (the black dotted box) of the L × L aerial image, the query and reference images are defined as positive samples for each other. Other queries (blue stars) outside the central area are defined as semi-positive samples. To guarantee that any arbitrary query has one positive reference image, we propose to densely sample the aerial images with 50% overlap along both latitude and longitude directions as demonstrated in Fig. 1 (c). By doing so, any arbitrary query location (the red star) in the AOI is covered by four reference images (size L × L). The green box denotes the positive reference and the other three semi-positive references are denoted as blue boxes. The positive reference is considered as ground-truth, because it has the nearest GPS to the query and contains the most shared objects with the query image. The red box denotes the perfectly aligned aerial image. Based on the definitions of positive and semi-positive as illustrated in Fig. 1 (b), we can easily see that all positive reference images have an IOU (Intersection Over Union) greater than 0.39 with the perfectly aligned reference (see Fig. 1 (d)). The IOU of a typical positive sample (offset relative to the center equals to (± L 8 , ± L 8 )) is 0.62. The IOU between the semi-positive samples and the aligned reference falls in [ 1 7 ≈ 0.14, 9 23 ≈ 0.39]. the distributions are included in the supplementary material). This procedure results in 105, 214 panoramas for the geo-localization experiments. Also, around 4% of the aerial images cover no panoramas. We keep them as distraction samples to make the dataset more realistic and challenging. The zoom level for satellite images is 20 and the ground resolution is around 0.114 m. The raw image sizes for aerialview and ground-view are 640 × 640 and 2048 × 1024, respectively. Industrial-grade GPS tags for both aerial-view and ground-view images are provided for meter-level evaluation. The panoramas are then shifted according to the orientation information so that North lies in the middle. Fig. 3 shows an example of orientation correspondence between a pair of aerial and street-view images. Head-to-head Comparison. Table 1 shows a comparison between our dataset and previous benchmarks. The most widely used dataset, CVUSA [27], consists of images mainly collected at suburban areas. Our dataset, on the other hand, is collected for urban environments. In practice, the GPS signal is more likely to be noisy in urban areas than suburban (e.g. the phone-based GPS error can be up to 50 meters in Manhattan [4]). Therefore, our dataset has more potential application scenarios, e.g. vision-based Vo [24] CVACT [11]  mobile assistive navigation. Besides, urban areas are more crowded with tall buildings. The mutual semantics between ground and aerial views are significantly reduced by occlusions and shadows, making our dataset more challenging than CVUSA. Furthermore, previous datasets simply adopt one-to-one retrieval for evaluation, which is not the case of real-world scenarios, because it is impossible to predict the location of an arbitrary query and capture an aligned reference image there beforehand. Our dataset considers arbitrary query locations, and even the ground-truth reference image does not have the same GPS location as the query; thus it is more realistic but challenging for retrieval. Our dataset also provides the raw GPS data for meter-level evaluation which is the ultimate goal of localization applications. We believe that our dataset is a great complement to the existing cross-view image datasets, and can be served as a testbed for bridging the gap between current research and practical applications. The Evaluation Protocol. We design two evaluation settings for the experiments, i.e. same-area and cross-area evaluation, according to different application scenarios. Same-area: If one plans to build an aerial-view reference database for arbitrary street queries in an AOI, the goal of model training is to handle arbitrary new queries. Therefore, the best solution would be collecting GPS-tagged queries in the same area for training rather than training in other areas with cross-area transfer. In this case, the aerial images in four cities are all included as the reference data for both training and testing. Then all the street panoramas are randomly split into two disjoint sets (see Table 2). Cross-area: For cities where no GPS-tagged queries are available for training, the cross-area transfer is necessary.

Manhattan
For this setting, all the images from New York and Seattle are used for training, and images from San Francisco and Chicago are held out for evaluation.

Coarse-to-fine Cross-view Localization
In this section, we propose a joint-retrieval-andcalibration framework for geo-localization in a coarse-tofine manner. Sec

Baseline Framework
To achieve satisfactory results on the proposed dataset, it is important to adopt state-of-the-art techniques to build a strong baseline. Therefore, we employ the feature aggregation module of SAFA (spatial-aware feature aggregation) [17] with the global negative mining strategy from [28]. Feature Aggregation. SAFA [17] is a combination of polar transformation, Siamese backbone and feature aggregation blocks. However, the polar transformation assumes that the ground-view GPS is at the center of the corresponding aerial-view reference image, which does not apply in our case. Therefore, we only adopt the feature aggregation in our framework (see Fig. 4). The main idea of the feature aggregation block is to re-weight the embeddings in accordance with their positions. The spatial-aware block provides a significant performance gain when the orientation information of query images is available. Mining Strategy. Metric learning literature [15,20,13] has revealed the importance of mining hard samples during training, as the model would suffer from poor convergence when most samples barely contribute to the total loss. For cross-view geo-localization, [28] further shows the importance of mining global hard samples instead of mining within a mini-batch. The key idea is to build a first-in-first-  out mining pool to cache the embedding of the hardest sample and refresh the pool along with back propagation efficiently. In a mini-batch, the first half images are randomly selected and the global hard samples with respect to each of them are mined from the mining pool to form the other half of the batch. We adopt this efficient global mining strategy [28] in the baseline to further improve its performance.

IOU-based Semi-positive Assignment
If we only consider the positive samples, the retrieval problem can be tackled with standard metric learning. For the baseline, we adopt the widely used triplet-like loss proposed in [7]: where d pos and d neg denote the squared l 2 distance of the positive and negative pairs. In a mini-batch with N groundview and aerial-view image pairs, we use the exhaustive strategy [15] to build 2N (N − 1) triplets, thereby making full use of all the input images. Following [7], we adopt l 2 normalization on the output embedding features. In addition to positive samples, it can be beneficial to take advantage of the supervision information of semipositive samples. However, simply assigning semi-positive samples as positive would hurt the retrieval performance. For a street-view query, the semi-positive aerial images only contain a small part of the scene at the query location, thus the similarities in the feature embedding space between semi-positive samples and the query should not be as high as those of positive samples. An intuitive idea is to assign the similarity according to the IOU between the reference image and the aligned one (see Fig. 1 (d)). Therefore, the IOU-based semi-positive assignment loss is expressed as: where S pos and S semi denote the cosine similarity of the positive and semi-positive pairs in the embedding space.
IOU pos and IOU semi denote the IOU of positive and semipositive pairs. This loss forces the ratio of the similarities in the embedding space to be close to the ratio of IOUs. Other assignment strategies for the semi-positive samples are also investigated and evaluated in the ablation study.

Offset Prediction
With the top-1 retrieved reference aerial image, we employ an auxiliary task to further refine the localization inside the aerial-view image in a unified framework (see Fig.  4). With image retrieval, the minimal interval between retrieved reference images in our dataset is half of the width of aerial images (L/2). To achieve more fine-grained localization, we apply an MLP (Multilayer Perceptron) to predict the offset of the query location relative to the center of the retrieved reference image. As shown in Fig. 4, the auxiliary MLP consists of two fully connected layers and takes the concatenated embedding features as input. Here we use regression to generate the prediction, while we also provide a comparison with classification in the experiments. The offset regression loss is formulated as: where lat and lon denote the predicted offset of the query GPS location relative to the reference GPS in latitude and longitude directions, and lat * and lon * denote the groundtruth offset. They are all converted into meters and normalized with L during training. The final hybrid loss function is given by:

Experiments
Implementation Details. All the experiments are deployed based on Tensorflow [3]. Ground-view panoramas and aerial-view images are resized to 640 × 320 and 320 × 320 respectively before being fed into the network. VGG-16 [19] is adopted as the backbone feature extractor and 8  Table 3. Retrieval accuracy (percentage) of different methods.  SAFA blocks are used by following [17]. The mining strategy parameters are set the same as in [28]. Following [7], we set α in the L triplet loss to 10. The Adam optimizer [8] is used with a learning rate of 10 −5 . Our method is trained for 45 epochs for the same-area setting and 20 epochs for the cross-area setting. The baseline (Sec. 4.1) for comparison is only trained with L triplet . Evaluation Metrics. We first evaluate the retrieval performance with the top-k recall accuracy following previous works [7,17]. For each test query, its closest k reference neighbors in the embedding space are retrieved as prediction. One retrieval is considered correct if the ground-truth image is included in the top-k retrieved images. If the retrieved top-1 reference image covers the query image (including the ground-truth), it is considered as a hit and the hit rate is also provided for retrieval evaluation. Moreover, we compute the real-world distance between the top-1 predicted location and the ground-truth query GPS as meterlevel evaluation. Main Results. On the proposed VIGOR dataset, we compare the proposed method with previous approaches under both same-area and cross-area settings. "Siamese-VGG" [7] is a simple Siamese-VGG network with global average pooling, and is trained with L triplet . "SAFA" and "SAFA + mining" denote the SAFA [17] architecture w/o and w/ the mining strategy in [28] using L triplet . As shown in Table 3 and Fig. 5, the proposed method constantly outperforms previous approaches in terms of both retrieval and meter-level evaluation. Compared with "SAFA+Mining" (the baseline), the relative improvements of our method for the 10-meter-level accuracy (see Fig. 5) are 124% (11.4% → 25.5%) in the same-area setting, and 121% (2.8% → 6.2%) in the cross-area setting. The substantial improvements reveal the superiority of the proposed hybrid loss.

Ablation Study
Semi-positive Assignment.
We compare four semipositive assignment strategies. "No semi-positive" denotes the baseline which ignores the semi-positive samples. "Positive" means assigning semi-positive samples as positive and using L triplet (Eq. 1). "IOU" denotes our IOU-based assignment (Eq. 2). "Positive+IOU" means including the semi-positive samples as positive in L triplet along with the IOU-based assignment loss. The results in Table 4 show that only IOU-based assignment ("IOU") boosts the per-  Table 5. Retrieval accuracy (percentage) of the proposed method with different offset prediction schemes. formance compared with the baseline. Based on the results of "Positive" and "Positive+IOU", assigning semipositive samples as positive hinders the retrieval performance whether the IOU-based loss is used or not.
To further illustrate the difference between positive and semi-positive matching, we conduct visual explanation. Specifically, we use Grad-CAM [16] to show which regions contribute the most to the cosine similarity of the embedding features of two views. As presented in Fig. 7, for a query image, we select the ground-truth reference aerial image (i.e. positive) and a semi-positive image (the query GPS location lies at its edge area), and generate the activation maps of both views. For the positive matching case, the surrounding objects (buildings, roads and trees) are all available to provide high contribution to the similarity between two views. However, in the case of semi-positive matching, two views only share half of the scene and the building on the west of the query (around 90 • in panorama) does not contribute to the similarity, because it is not in this semi-positive image. This example shows how the intersection semantics of two views affect the image matching, which agrees with the design of our IOU-based assignment.
Offset Prediction. We investigate both regression and classification for offset prediction in our method. For classification, we split the central area of an aerial image (offset in [−L/4, L/4]) into a 10 × 10 grid, leading to 100 classes for classification. As shown in Table 5, both regression and classification have negligible improvement on the retrieval performance. However, as evident from Fig. 6, regression-based calibration significantly boosts the meterlevel accuracy in both settings. For example, the regression  . Case study on meter-level refinement within the retrieved aerial image. Red square, green circle and blue diamond denote the final prediction with regression, ground-truth, and center (i.e. the prediction with only retrieval), respectively. method almost doubles the 10-meter-level localization accuracy. However, classification does not work well for calibration possibly due to the ambiguous supervision of gridbased classification. We provide a case study in Fig. 8 to show examples of predicted offset on aerial images.  Table 6. Retrieval accuracy (percentage) of the proposed method with different sample balancing settings. Sample Balancing. To investigate the effect of sample balancing in the pre-processing procedure, we compare "balancing-2" with "balancing-3" in Table 7. The results in Table 6 show that more densely sampled panoramas bring slightly better performance on both settings, while the improvement is consistent across different balancing settings.

Application: Assistive Navigation
The GPS data provided by commercial devices such as phones could be noisy in urban environments (e.g. the phone-based GPS error can reach up to 50 meters in Manhattan [4]). Image geo-localization can assist mobile navigation. To further validate the potential of crossview geo-localization given noisy GPS signals [4,26], we simulate noisy GPS signals by adding random offsets in [−100m, 100m] to the ground-truth GPS data (latitude and longitude) in our dataset. In the inference stage, the query image can be matched with only a small sub-set of reference images around the noisy GPS locations instead of the entire reference database (denoted by "All"). For a noise level of 100m, a search scope of 200m is sufficient to cover all possible references. To better illustrate the navigational assistance provided by our image geo-localization, we compare the results of multiple scopes with simply using the noisy GPS signals ("Original"). As shown in Table 8 and Fig. 9, smaller search scopes generate better results for both retrieval and meter-level evaluation because there are less negative reference samples. The same-area evaluation even yields an accuracy higher than 70% for 30m-level localization. Moreover, as compared to the original noisy GPS, our cross-view geo-localization method significantly improves the localization accuracy, demonstrating its practicality in real-world applications.

Conclusion
We propose a new benchmark for cross-view image geolocalization beyond one-to-one retrieval, which is a more realistic setting for real-world applications. Our end-to-end framework first coarsely localizes the query with retrieval, and then refines the localization by predicting the offset with regression. An IOU-based hybrid loss is designed to leverage the supervision of semi-positive samples. Extensive results show great potential of the proposed method in realistic settings. Our proposed dataset offers a new testbed for cross-view geo-localization and inspires novel research in this field.