{'fraction_non_alphanumeric': 0.042244046818242974, 'fraction_numerical': 0.018512040898695008, 'mean_word_length': 4.584673178061608, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Beyond Controlled Environments: 3D Camera Re-Localization in Changing Indoor Scenes

Long-term camera re-localization is an important task with numerous computer vision and robotics applications. Whilst various outdoor benchmarks exist that target lighting, weather and seasonal changes, far less attention has been paid to appearance changes that occur indoors. This has led to a mismatch between popular indoor benchmarks, which focus on static scenes, and indoor environments that are of interest for many real-world applications. In this paper, we adapt 3RScan - a recently introduced indoor RGB-D dataset designed for object instance re-localization - to create RIO10, a new long-term camera re-localization benchmark focused on indoor scenes. We propose new metrics for evaluating camera re-localization and explore how state-of-the-art camera re-localizers perform according to these metrics. We also examine in detail how different types of scene change affect the performance of different methods, based on novel ways of detecting such changes in a given RGB-D frame. Our results clearly show that long-term indoor re-localization is an unsolved problem. Our benchmark and tools are publicly available at waldjohannau.github.io/RIO10


Introduction
Visual re-localization is the problem of estimating the precise position and orientation from which a given image was taken with respect to a known scene. It is a key component of advanced computer vision applications such as AR/VR [8,22,42,65,73,77,101], and robotics systems such as self-driving cars [19] and drones [63]. Real-world scenes are highly dynamic, exhibiting changes in illumination, appearance and/or geometry. These changes are caused by a variety of factors, including the time of day, the presence of artificial light sources and, most prominently, humans interacting with their environments, e.g. by redecorating a room, or using furniture and objects in day-to-day life. Like human perception, visual re-localization algorithms should be as robust as possible to such changes to enable long-term operation in the real world.
Train Setup: Test Setup: Fig. 1. Visual re-localization in changing indoor scenes: we introduce a new benchmark based on 3RScan [104], together with a new evaluation methodology, for measuring 6DoF re-localization performance given a reference RGB(-D) sequence of an indoor scene at time T0 (left), and query sequences taken at different points in time (center and right).
However, the datasets traditionally used for evaluating visual re-localization performance [27,46,51,61,62,85,88,99] either do not contain such changes [51,88,99] or do not provide a means of quantifying their impact [27,46,61,62,85,92]. Only recently released datasets such as Aachen Day-Night [84,85], (extended) CMU Seasons [7,84], RobotCar Seasons [66,84], and SILDa [9] explicitly model such changes. By providing a reference representation and test images taken under different conditions, the corresponding works point out failure cases of existing re-localization algorithms, in turn motivating the community to devise more robust methods [2,11,34,38,53,75,80,94,95]. However, these datasets mostly focus on outdoor scenes, where most changes are cyclic (e.g. day-night, seasonal and weather changes) and can thus easily be predicted by neural networks [2,75]. As shown in Fig. 1, indoor scenes are arguably more diverse, and exhibit changes -including complex illumination changes, as well as geometric and appearance variations caused by human interaction -that are harder to predict. The only indoor datasets exhibiting these types of changes [21,92] were captured in public spaces, where there is limited human interaction with the environment. Just as importantly, these datasets do not quantify changes, and do not provide the means to measure their impact on re-localization performance. They therefore cannot be used to measure to what degree existing re-localization algorithms are able to handle realistic changes occurring in everyday indoor scenes. This paper makes the following contributions: (1) We construct an indoor re-localization benchmark based on a recently released dataset, 3RScan [104]. 3RScan captures everyday scenes over a long period of time (c.f . Fig. 1) and thus depicts a wide range of changes not captured by other datasets from the literature. (2) We propose a novel framework to quantify changes in (indoor) scenes, covering appearance, geometric, and semantic changes. This enables us, for what to the best of our knowledge is the first time, to quantifiably measure the impact of different types of change on the accuracy of the camera poses predicted by visual re-localization algorithms. (3) We evaluate state-of-theart methods for re-localization in static indoor and changing outdoor scenes, and show through detailed experiments that indoor re-localization in real-world scenes is far from being solved. (4) Based on our experiments, we propose a set of open challenges for the community to work on. We make our benchmark, framework, and evaluation protocols publicly available. We think this benchmark closes a gap in the literature by going beyond controlled indoor environments, similar to recent high-impact benchmarks modelling outdoor scene changes [7,66,84,85].

Related Work
Benchmarks. A variety of datasets exist to target different aspects of the camera re-localization problem (c.f . Tab. 1) 1 . For the task of re-localizing in outdoor scenes that change over time, a multitude of benchmarks exists that look at day vs. night changes, season and weather changes, and long-term geometric changes based on e.g. changing vegetation or construction projects. Aachen Day-Night [84] extends the Aachen dataset [85] to support evaluation of a re-localizer's ability to estimate the poses of night-time, outdoor, RGB-only images against a day-time 3D model. RobotCar Seasons [84] is based on a subset of the outdoor Oxford RobotCar dataset [66]. It focuses on re-localization across different seasons and weather conditions, but also contains a challenge related to localizing lowquality night-time images. While RobotCar Seasons covers an urban region, the (extended) CMU-Seasons dataset [7,84] also covers more vegetated outdoor scenes. SILDa [9] depicts a small block of buildings in London and provides test images under changing conditions such as weather and illumination changes.
For many years, the most popular indoor datasets have been 7-Scenes [88] and 12-Scenes [99], which only contain static scenes and exhibit no changes between train and test time. There do exist indoor datasets containing changes, e.g. InLoc [92] consists of non-sequential RGB-D training images that are registered to floor plans of university buildings [106], and RGB-only query images taken at a later date by hand-held devices. Moreover, InLoc and NCLT [21] both contain scene changes such as moved objects. However, neither provide any means of quantifying the impact that different changes have on re-localization performance. In this paper, we address this problem by introducing a novel framework to properly quantify the effects of changes in indoor scenes.
Camera re-localization methods can be broadly divided into four types: Image retrieval methods typically match the query image against images with known poses in a database [36,40], but can struggle to generalise to novel poses. Strategies to mitigate this include the use of synthesized views [37,96], interpolation between database poses [10,54,97,109], and triangulation based on relative poses [110,111]. To achieve scalability in terms of memory and run-time, place recognition methods [3,96] typically use compact image-level descriptors. Such methods perform well under appearance and limited viewpoint changes [84]. Direct pose regression methods, which aim to directly regress a pose from the query image, are often based on pose regression networks [1, 49-51, 68, 107], although decision forest [47], GAN [20] and LSTM [29,103] variants also exist. On the whole, they have not yet matched the precision of state-of-the-art structure-based and RGB-D methods indoors. Recent work by Sattler et al . [86] has suggested that they are conceptually similar to image retrieval, and may thus face ongoing challenges in generalising to novel poses and achieving highly accurate pose predictions. Some direct pose regression methods [18,57,76,98] now exploit the relative poses between images to improve accuracy, and in some cases [76,98] have achieved accuracies that are competitive with state-of-the-art RGB-D methods. However, thus far they have had to rely on estimated poses from previous frames, making them effectively camera tracking approaches that are incomparable with methods that are able to re-localize from only a single image. Structure-based methods typically match 2D features in the image with 3D points in the scene, and then pass the correspondences to a RANSAC-based backend for camera pose estimation. A classic example is Active Search [83], which performs efficient bidirectional matching using SIFT-based visual vocabularies. Hierarchical localization methods [46,80,82,92,93] use an initial image retrieval step to make matching more efficient, i.e. they first determine a set of potentially visible locations and restrict 2D-3D matching to these. For long-term localization under changing conditions, state-of-the-art methods typically rely on learned features [34,38,80,105], e.g. HF-Net [80] uses sparse SuperPoint [32] and DOAP [44] features, whilst [34] uses sparse higher-level features extracted from deeper layers of a CNN. Both achieve state-of-the-art results on outdoor benchmarks from [84] and outperform approaches based on dense feature matching [38,92,105]. # Changed Object Instances 5-9 5-6 2-3 1-5 1-2 1-6 6-10 1-5 5-6 7-9 Another popular approach to outdoor long-term localization is to use semantic information [53,87,94,95]. However, [93] argues that most of these approaches are not directly applicable to indoor scenes. Similarly, object-based localization methods [4,6,56,79] do not seem applicable in the context of re-localization in changing indoor scenes, as many objects are likely to change their position. Scene coordinate regression (SCoRe) methods densely regress the scene coordinates of query image pixels using a regression forest [13,[23][24][25]41,43,[69][70][71]88,100], a neural network [12, 14-17, 33, 59, 60, 108], or both [67]. The correspondences are used to generate pose hypotheses using PnP/Kabsch that are then refined using RANSAC. These methods can be categorised based on whether they expect RGB [12-16, 33, 59, 60, 69, 70, 108] or RGB-D [23-25, 43, 71, 88, 100] input at test time, and whether they require offline training (most methods) or can be used online [23][24][25]. Better performance has typically been achieved using RGB-D [24] rather than RGB-only [14,15] input, although RGB-only methods are gradually closing the gap. The state-of-the-art SCoRe relocaliser for indoor RGB-D scenes is currently Grove v2 [24], an online regression forest method, although a network-based variant of this [23] performs better outdoors. A few approaches defy such a categorisation. Valentin et al . [99] use continuous pose optimisation to refine the results of an initial matching process based on a retrieval forest and multiscale navigation graph. Nakashima et al . [72] replace the feature matching step in hierarchical localization with dense regression. Other methods perform retrieval using a point cloud [31] or 3D model [64] constructed from multiple query images, or hallucinate a subvolume and match that against a database [87]. Since our main contribution here is to propose a new benchmark and metrics for evaluating camera re-localization in changing indoor scenes, we focus our attentions on those re-localizers that are known to currently have state-of-the-art performance on static indoor scenes or dynamic outdoor scenes, and explore how their performance is affected when the scenes change.

Benchmark Dataset
The original 3RScan [104], which was the first large-scale, real-world dataset of changing indoor environments, consists of 1482 3D scans of around 450 nat-ural indoor environments. Each scene has m globally aligned 3D models, each reconstructed from an RGB-D sequence s recorded at time T s , using a hand-held Google Tango phone with camera intrinsics K s ∈ R 3×3 . Reasonably accurate camera poses {P s,1 , ..., P s,ks } for each sequence s (of length k s ) are determined via an offline bundle adjustment framework, based on fisheye images. A pose is defined by a rotation matrix R s,i ∈ R 3×3 and a translation vector t s,i ∈ R 3 . Note that P s,i transforms from the local camera coordinate system to the 3D model coordinate system. Whilst originally designed for object re-localization, 3RScan can also -when slightly adapted -enable benchmarking of related tasks such as long-term camera re-localization. Due to the large size of the original dataset, we have chosen to focus on a 10-scene subset of it, which we call RIO10, for our experiments and evaluation protocol (c.f . Tbl. 2). We split the sequences and 3D models into training, validation (one sequence per scene) and testing sets, leaving us with 10 train, 10 validation and 54 test sequences overall. The provided 3D models have both color and semantics (see Fig. 2), and are defined as {M s : 0 ≤ s < m}, where M 0 is our reference/training scan, and each other scan M s is a test or validation scan. The 10 scenes chosen for RIO10 were selected due partly to their scanning frequency, and partly to their scene and change diversity. Indeed, they are among the scenes in 3RScan with the highest time span and scanning frequency (5-12 scans each). RIO10 features many different indoor scenarios (messy laundry basements, offices or bathrooms) and different types of change (e.g. diversity in lighting, both subtle and significant movements of large/small and rigid/non-rigid objects, and ambiguous changes where objects of the same appearance move). Whilst we decided to evaluate on only a small subset of 3RScan, the remaining scans are still useful for training future models.
To simplify evaluation, each test 3D model and camera pose is provided in the training sequence's reference frame. Due to the low resolution and frame-rate of the raw Tango depth maps, we generated depth renderings of the 3D models for each RGB frame, together with ground-truth 2D instance segmentations. For reproducibility, all data, along with the evaluation tools and per-frame statistics, will be made publicly available.

Evaluating Re-Localization in Changing Indoor Scenes
Having described our benchmark dataset, we now propose an evaluation methodology for the well-studied camera re-localization problem, as well as novel ways to quantify scene changes. Compared to common evaluation measures from previous camera re-localization benchmarks, we show the advantages of alternative metrics such as the normalised absolute correspondence re-projection error (Sec. 4.2) when measuring camera re-localization performance. To analyse how relocalization methods are able to generalise to changes in the scene, we propose various measures to quantify the change in each image. This is important, since it gives us an understanding of whether and how different methods are affected by different types of scene change.

Quantifying Change in (Indoor) Scenes
In the following, we introduce different measures to quantify the extent to which an RGB-D frame in one of the test sequences has changed with respect to the same view of the reference scan. To compute the measures, we make use of synthetic views of the globally aligned semantic and textured 3D models in 3RScan (see Sec. 3). To produce these synthetic views, we define three different rendering functions: R C for color, R D for depth, and R S for semantics. Each of these takes a 3D model M, a pose matrix P ∈ R 4×4 , and a camera intrinsics matrix K ∈ R 3×3 , and produces a w × h synthetic view of M as seen from P using a camera with intrinsics K (see Fig. 2

for examples).
Visual Appearance Change Given these rendering functions, we can define measures for the visual appearance change between two different models M and M as seen from a given pose P by a camera with intrinsics K. Let I = R C (M, P, K) and I = R C (M , P, K) be color renderings of the two models from P . Given these, we consider two different measures of the visual appearance change -the normalized correlation coefficient ρ v , defined as and the normalized sum of squared differences ζ v , defined as in whichĪ(u) = I(u) − 1 w·h u I(u ). Note that in our experiments, M is a rescan (see Fig. 2(e)), M is the corresponding reference scan (see Fig. 2(a)), and P is a pose from one of the rescan/testing sequences.
Semantic Change We can also define a semantic change measure ζ s , based on the percentage of altered pixels in the 2D instance segmentation images. Let L = R S (M, P, K) and L = R S (M , P, K) be semantic renderings of the two models from P , and V (s) {L,L } be the set of pixels that have a valid instance ID in both L and L . Then we can define {D} . Then we can define We report ζ g as a value in millimeters. Note that this measure would be particularly high for the depth renderings from pose P j in Fig. 2, since in one of the models, a door has been moved so as to block the view.
Change Statistics Please note that change statistics for each scene can be found in the supplementary material.

Measuring Re-Localization Performance
Given a sequence of ground truth poses as 3D orientations {R 1 , ..., R p } (where R i ∈ R 3×3 ) and absolute 3D locations {t 1 , ..., t p } (with t i ∈ R 3 ), as well as corresponding pose estimates {R 1 , ...,R p } and {t 1 , ...,t p }, common evaluation protocols are based on absolute pose errors. More specifically, it is common to compute the absolute translation error as a Euclidean distance in meters, namely ∆t i = ||t i − t i ||, and the absolute orientation error as an angle in degrees, ||, in which q(R) denotes the quaternion corresponding to the rotation matrix R. Methods can then be ranked by comparing their values for E a orĒ a , the fraction of images localized within (eq. 6) or outside of (eq. 7) the given error thresholds ( t , θ ):  Commonly chosen thresholds for E a in indoor setups are (0.05m, 5 • ) or (0.1m, 10 • ). However, these values are manually selected, and do not correlate with the visual appearance of a scene: a one-pixel shift could potentially lead to a pose error of only a few millimeters when objects are close, but a few meters if objects are far from the camera. Instead of using hard thresholds, [51] independently reports the medians ∆t and ∆θ of the absolute translation and angular errors. However, these median errors can correspond to completely different frames, and there is in fact no guarantee with this measure that any single frame has both a low translation error and a low angular error, even if both the medians are low. In this paper, we eschew both of these approaches and instead propose a new measure that, rather than being based on the absolute translation and angular errors, is directly based on the difference in appearance between an image from the ground truth pose and an image from the predicted pose.
Dense Correspondence Re-Projection Error (DCRE) Our new measure, which we call the Dense Correspondence Re-Projection Error, is defined as a ground truth re-projection error of the 2D flow of dense 3D points rendered from an underlying 3D model (see Fig. 3). The flow is computed according to our ground truth and predicted camera poses. Specifically, the 3D model for the sequence of interest s is first rendered from the ground truth pose P s,i . This gives us a high-resolution dense depth map D i = R D (M s , P s,i , K s ) that can be back-projected into a 3D point cloud using the back-projection function Π −1 Ks . The points in the cloud are then transformed byP −1 s,i P s,i before being projected back down onto the image plane using Π Ks to get a new depth map. The flow error δ (i) f at a pixel u in frame i can then be defined as Intuitively, the overall frame error E This can then be extended to a DCRE-based error E f ( f ) for the whole sequence: One major advantage of such a measure is that it gives us an error that correlates with visual perception (see examples in Fig. 4). Another desirable property of DCRE is the fact that it is represented by a single number, which is in contrast to absolute pose errors, which struggle to combine the translation error ∆t i with the angular error ∆θ i . Furthermore, a (cumulative) DCRE histogram can provide us with a good way of characterising the performance of a method (see Sec. 5.2), since it represents the poses within a wide error range.

Experiments
To evaluate the impact of appearance changes on indoor camera re-localization, we analyse the performance of state-of-the-art re-localizers on RIO10 using both common evaluation metrics and our newly proposed DCRE measure (Sec. 4.2). We also conduct experiments to evaluate how robust different re-localizers are with respect to various types of change, as suggested in Sec. 4.1.

Classifying Frame Difficulty
Scene changes are one factor that can make single-image re-localization challenging, but other factors (e.g. scene context and texture, or the pose novelty with respect to the training trajectory) can also play a significant role. We thus propose to rank the difficulty of each query image based on the following three properties. More details can be found in the supplementary material.
Variance of Laplacian Many feature-based methods struggle when confronted with motion blur and a lack of texture. To be able to detect such images, we compute the variance of the Laplacian of the image, which we refer to as σ.
Field of View Context Besides a lack of texture, a lack of scene context can present another major challenge for camera re-localizers. To estimate the field of view of a particular frame, we first back-project the depth map. The volume of the convex hull of the resulting 3D points, combined with the camera center, gives an estimate of the context observed in a particular view (see Fig. 5).
Pose Novelty Another major challenge for camera re-localizers is the novelty of query poses with respect to the training trajectory. Given a sequence of poses {P 0 , ..., P p } from the train set, and a ground truth query pose P , we can define the pose novelty η as the minimum of some dissimilarity function η between all pose combinations, such that η = min ∀P i ∈{P 0 ,...,P p } η (P, P i ).

Re-Localization Performance
In the following, we evaluate a selection of state-of-the-art algorithms that cover the most common types of re-localization approach: hand-crafted structure-based methods [81], learned methods 2 that expect either RGB [34,80] or RGB-D [24,25] input, and image retrieval methods [3,96]. In our first experiment, we evaluate on all 165 744 query test images, without any filtering. We list the overall performance of each method in Tbl. 3 by reporting E f (0.05) and E f (0.15), based on our newly introduced DCRE metric. For comparison, we also report the recall based on the absolute pose error E a , with the often-used thresholds ( t , θ ) = (0.05m, 5 • ). Further, we also quantify the number of re-localization outliers 3 , by reporting both the percentage of frames with a high E a error, with ( t , θ ) = (0.5m, 25 • ), high DCRE error, with E f (0.5), and failed re-localizations (no predicted pose or NaN). Single numbers are still not really descriptive of the dynamics of each  algorithm. We thus visualize cumulative plots in Fig. 6 using DCRE, as well as ∆t i and ∆θ i for comparison. These graphs shed some light on the behavior of the methods that we analyze. For example, it is interesting that the best-performing methods according to the threshold-based metrics E a and E f , such as Grove v2 and D2-Net, output increasingly inaccurate poses, as evidenced by the steady increase in their DCRE values towards the right of the plot. By contrast, Active Search tends to provide poses for a smaller number of query frames but, crucially, does not output overly incorrect poses, as evidenced by the plateauing of its DCRE plot. While some of this information can also be gained by analysing the numbers in Tbl. 3, we find that the cumulative plot provides a deeper, more intuitive characterisation of each method. An ideal method should yield a cumulative DCRE that is as similar to a step function as possible: first rising quickly to correctly re-localize a good fraction of the frames, and then plateauing (signalling failed re-localizations instead of producing highly incorrect poses).
Scene Changes To see how scene changes affect a method's performance, we plot the overall error/performance of the best methods with images of increasing visual (ζ v and ρ v ), geometric ζ g and semantic change ζ s . A clear correlation between scene changes and overall performance is observable in Fig. 7.  Object Re-Localization vs. Camera Re-Localization Rigidly moving objects cause new types of absolute camera pose estimation ambiguities. Poses become ambiguous when a changed object occupies most of the view. An example is given in Fig. 8, where localizing the test image (c) from the rescan (b) is practically impossible given only the reference scan (a). The correct reference view of the GT pose would produce the reference view pictured in (d). Instead, when an object instance dominates the view, the camera might incorrectly localize with respect to the visible object. We report the fraction of these cases (out of all failure cases) in the last column of Tbl. 3.

Sequences
We experimented with sequence lengths s ∆ of (a) 10, (b) 30, and (c) 100 consecutive frames. The corresponding DCRE plots can be found in Fig.  9. We chose the values 10 and 30 to model interactive applications, where using a small number of consecutive frames can help tackle motion blur and object instance ambiguities; whereas longer sequences of up to 100 frames can be used in less time-sensitive applications, where re-localization accuracy is more important than interactivity. As the figure shows, when leveraging frame sequences, there is a significant improvement in the DCRE numbers.  Fig. 9. Cumulative plots of the DCRE for the best-performing camera re-localization methods using (a) short, (b) medium, and (c) long sequences of frames.

Conclusion
In this paper, we have both curated a suitable dataset for long-term indoor camera re-localization, and defined a set of metrics for quantifying changes in indoor scenes. For the first time, this enables an evaluation of the impact of changes in indoor scenes on re-localization performance, thus closing a significant gap in the literature. We have also introduced DCRE, a new metric to measure re-localization performance, and shown that many methods experience a loss of performance when exposed to scene regions that have undergone changes of visual, geometric and semantic nature, e.g. as caused by rigid/non-rigid object movements. We have further analysed the behaviour of camera re-localizers on frames that capture rigidly moving objects. Large semantic changes, e.g. caused by large objects in a scene changing their position, are a particular problem. In such situations, the methods potentially re-localize with respect to the object dominating the camera's field of view, rather than with respect to the scene. Results for state-of-the-art re-localizers on our new benchmark show that none of them is fully capable of handling everyday changes observed in indoor scenes: indeed, there is significant room for improvement. Using short image sequences, rather than individual images, for re-localization naturally improves performance, but is not sufficient to solve our benchmark. We believe that long-term camera re-localization in indoor scenes requires the learning of higher-level concepts of a scene -such as its semantics, and/or object-level understanding of poses, dynamics and appearance variations -so as to subsequently be able to reason about scene changes. In this way, we would expect the camera pose estimation task to gradually become more tightly coupled to general scene understanding going forwards.

Supplementary Material
This supplementary material provides the following information: Sec. 7.1 shows the individual scenes of our benchmark dataset. Sec. 7.2 provides statistics about the changes that occur in our benchmark dataset (c.f . Sec. 4.1 in the main paper). Sec. 7.3 discusses the means by which we classify the difficulties of the test frames in our dataset (c.f . Sec. 5.1 in the main paper). Sec. 7.4 provides further details about our DCRE metric (c.f . Sec. 4.2 in the main paper). Sec. 7.5 discusses implementation details for HF-Net and the image retrieval methods we tested (c.f . Sec. 5.2 in the main paper), as well as the sequence-based approaches evaluated in the main paper. In addition to this document, we also provide a supplementary video summarizing our paper.

Benchmark Visualization
Figs. 15 -24 show the 3D reconstructions of each of the individual scenes in the RIO10 dataset. The scenes selected for RIO10 are very diverse, and exhibit a wide variety of changes, including, but not limited to, complex illumination changes, and appearance variations mostly caused by human interactions, such as rigid object movements (e.g. the movement of major objects such as the bed and sofa in scenes 3 and 4, respectively) and non-rigid object deformations (e.g. the rearrangement of the blankets in scene 3). Our dataset provides 10 train, 10 validation and 54 test sequences, with 52 562 images in the train set, 34 415 images in the validation set and 165 744 in the test set.

Change Statistics
Per-scene change statistics corresponding to the change measures described in Sec. 4.1 of the main paper can be found in Fig. 10. It can be seen that on the one hand, scene 4 has the highest semantic (c) and geometric (d) change values (many objects, including a sofa, move in the rescans), whilst on the other hand, scenes 8 and 9 have a low normalised correlation coefficient (a) and a high normalised SSD (b), highlighting the visual differences that they contain (see also the original scans in Figs. 18, 22 and 23).

Classifying Frame Difficulty
Variance of Laplacian (VoL) As mentioned in the main paper, the Variance of Laplacian (VoL) measure captures both motion blur and a lack of texture in an image. Fig. 11 shows the average of this measure over all frames from all test sequences for each scene. Blurred images, or images with a lack of texture, such as the left two images in the figure, have a low VoL value and often lack features needed for localization, which can sometimes make them difficult even for humans to localize. By contrast, images with a higher VoL value, such as the right two images in the figure, often contain more descriptive features and are  therefore expected to be easier for feature-based re-localization algorithms such as Active Search [81] to handle.
Pose Novelty Fig. 12 shows a selection of test images, together with their nearest neighbours in the corresponding training sequences, as computed by using our novel DCRE measure as a pose similarity metric. Image pairs with higher DCREs broadly correspond to test images that were captured from more novel poses.
Field of View/Context The left side of Fig. 13 shows the average field of view/context for a test frame in each scene of our RIO10 dataset, as per the description of this metric in Sec. 5.1 of the main paper. On the right side of Fig. 13, some example test images with a context of > 10m 3 are shown. In our experiments in Table 6, it can be seen that methods struggle with low-context frames (compared to medium-context ones). Interestingly, our high-context frames proved more challenging than our medium-context ones on average, potentially due to a combination of factors such as motion blur, lack of texture and large scene element changes in some of our high-context frames.

Dense Correspondence Re-Projection Error
DCRE is a dense re-projection error of ground-truth correspondences applied in a novel context, namely the evaluation of camera re-localization. Compared to traditional applications (e.g. camera calibration or bundle adjustment), synthetic depth images are used, which gives us ground-truth correspondences. The DCRE will thus be 0 for the ground-truth pose. By contrast, the reprojection errors for SfM point clouds will generally be non-zero due to noise in the image measurements. The point clouds generated by rendering a 3D mesh further enable us to compute the metric densely over the whole image rather than being restricted to well-textured regions in the images where features are extracted. Despite its advantages, we are not aware of any re-localization benchmark that uses dense re-projection errors for evaluation.
Please note that we decided to normalize the DCRE error (see eq. 9). While normalization is not strictly necessary for this dataset, as all images have the same resolution, it helps future research to compare errors across datasets. 5px is not much in a 5K image, but might be significant at lower image resolutions.

Implementation Details
Details for HF-Net and Image Retrieval We trained HF-Net for 50k iterations on the 52k training images of RIO10. Image retrieval interpolation results (based on the top 20-NN) achieve very similar performance (see Fig. 14 and Table 4). Implementation Details for Sequence-based Re-Localization The following provides details about the sequence-based re-localization experiments presented in Fig. 9 of the main paper. We use two different approaches, one for RGB-only (Active Search and D2-Net) and one for RGB-D methods (Grove and Grove v2). For both, a sequence is defined as a consecutive set of frames with known relative poses. For all our experiments, we use relative poses defined by the ground truth absolute poses. Note that this does not provide sequencebased methods with any information about where in the scene the images were taken, but it eliminates the impact of pose tracking errors, e.g. due to drift in visual odometry or SLAM, from the localization process. As such, the experiments presented in the paper represent an upper bound on the performance of sequence-based approaches. Closing the gap between this upper bound obtained with "perfect" relative poses and relative poses computed by an existing odometry/SLAM system remains an open research question. However, Fig. 9 in the paper shows that considerable gains are possible, which should make practical implementation of sequence-based re-localization an interesting research topic.
For RGB-only methods, we model a sequence of images with known intrinsics and relative poses as a generalized camera [74], i.e. as a camera with multiple centers of projection. The 2D-3D matches found for each individual image then allow us to estimate the pose of the generalized camera (i.e. of all images in the sequence simultaneously) by applying a minimal solver for the generalized perspective-n-point pose (gPnP) problem [52,55,91,102] inside a RANSAC [35] loop. More precisely, we use a gPnP+s solver [52] that estimates both the pose of the image trajectory and a scale factor, i.e., our approach could account for scale differences between the global 3D model and the trajectory.
For RGB-D methods that process and relocalize each frame independently, we adopt a different approach. Specifically, for each sequence we want to evaluate, we first transform the relocalized pose for each frame (which denotes the estimated transformation from that frame's camera pose to the origin of the reference scene) into a pose expressed relative to the last frame in the sequence. This computation is done by combining the frames' relative poses with the relocalization output. We then cluster the transformed relocalized poses (each of which denotes a possible transformation between the last frame's camera pose in the current scene and the origin of the reference scene) using an iterative approach. Typically, as a result of the clustering, there will be a single large cluster of poses that are mutually similar, and a number of outliers. We return, as the relocalization result for the sequence, the centroid of the largest cluster (computed, for robustness, via dual-quaternion blending [48] of the corresponding poses).
We will release the code for both of these approaches, thus enabling researchers to more easily work on sequence-based localization. Table 5. Filter setup for evaluation of different challenges on the test / validation images (see Table 6). In the following, ν is the field of view of a frame (as described in Sec. 5.1 of the main paper).  Table 6. Evaluation of the different camera re-localization methods with different setups (described in Table 5); the reported numbers are E f (0.15).