{'fraction_non_alphanumeric': 0.028690996833056853, 'fraction_numerical': 0.009727039662192731, 'mean_word_length': 4.964695300202384, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data Efficient Visual Place Recognition Using Extremely JPEG-Compressed Images

Visual Place Recognition (VPR) is the ability of a robotic platform to correctly interpret visual stimuli from its on-board cameras in order to determine whether it is currently located in a previously visited place, despite different viewpoint, illumination and appearance changes. JPEG is a widely used image compression standard that is capable of significantly reducing the size of an image at the cost of image clarity. For applications where several robotic platforms are simultaneously deployed, the visual data gathered must be transmitted remotely between each robot. Hence, JPEG compression can be employed to drastically reduce the amount of data transmitted over a communication channel, as working with limited bandwidth for VPR can be proven to be a challenging task. However, the effects of JPEG compression on the performance of current VPR techniques have not been previously studied. For this reason, this paper presents an in-depth study of JPEG compression in VPR related scenarios. We use a selection of well-established VPR techniques on 8 datasets with various amounts of compression applied. We show that by introducing compression, the VPR performance is drastically reduced, especially in the higher spectrum of compression. To overcome the negative effects of JPEG compression on the VPR performance, we present a fine-tuned CNN which is optimized for JPEG compressed data and show that it performs more consistently with the image transformations detected in extremely compressed JPEG images.


I. INTRODUCTION
Several robotic applications benefit from deploying multiple units operating in parallel, such as exploration and rescue missions where multiple robots can cover the search area easier than a single agent. Knowing their position in the operating environment as well as the positions of the other agents is fundamental for mobile robots. As part of the visual simultaneous localization and mapping (SLAM), visual place recognition (VPR) is an essential task for the localization process when the environment is unstructured, GPS is unavailable or the visual odometry drifts due to accumulated errors. For applications that involve a single robot, an ideal VPR method is accurate in detecting known places and efficient enough to fit the hardware and battery capability of the deputed robot [1]. For applications requiring with different compression percentages (0%, 50%, 80%, 90%, 95% and 97%). It can be seen that above 90% compression the image is drastically changed from the original uncompressed image. multiple robotic platforms to collaborate, navigate and map the environment effectively, the visual data gathered must be transmitted remotely between each robot [2], [3], [4]. Hence, the amount of data required to be transmitted must be taken into consideration when working with limited bandwidth available for VPR. This paper proposes to use highly compressed JPEG images to reduce the amount of data that is transmitted in decentralized VPR contexts. ISO/IEC-ITU JPEG is one of the most widely used compression standards employed to facilitate significant data storage and transmission reduction [5]. In our opinion, the high compression ratios enabled by JPEG compared to other standard techniques [6], [7] make it attractive for distributed VPR applications. However, JPEG is designed to have a minimal impact on the human perception system [8]. It is uncertain how the performance of various VPR techniques is affected throughout the compression spectrum. To the best of our knowledge, the study of JPEG for VPR application has been overlooked so far. In this paper, we propose to tune CNNs to deal with highly compressed JPEG images to circumvent the limitation of existing techniques. In summary, our contributions are the followings: • An assessment of several well-established VPR techniques under mild to extreme JPEG compression rate, as shown in Fig. 1. This analysis uses several datasets presenting illumination, viewpoint, and weather variations to cover some of the most common viewing conditions experienced by a robot in real-world decentralized applications, where the operating environment might present heterogeneous conditions in different places (see Fig.  4). • We propose a CNN-based descriptor trained to handle JPEG compressed data to achieve higher and more consistent VPR performance on highly JPEG compressed images than non-optimized VPR techniques. The remainder of this paper is organised as follows: Section II presents a literature review. In section III, JPEG compression is presented, together with the place matching approach employed and the performance metric utilised in conducting our analysis. Section IV describes the utilised datasets and VPR techniques for performing the analysis on the effects of JPEG compression in VPR. Section V presents the detailed results and analysis. Finally, the conclusions are presented in Section VI.

II. LITERATURE REVIEW
Environmental changes such as illumination [9] and viewpoint variation [10] make a place appear differently on different traverses. These appearance changes render visual place recognition a challenging problem motivating significant effort put by the research community in proposing improvements to existing VPR methods and new techniques. Local feature descriptors such as Speeded-Up Robust Features (SURF) [11] and Scale-Invariant Feature Transform (SIFT) [12], [13] have been widely used in VPR [14], [15], [16], [17], [18]. FAB-MAP [19] is an appearance based place recognition system that represents visual data as words and uses SURF for feature detection. Odometry information is included in FAB-MAP by the authors of [20] to create CAT-SLAM. Center Surround Extremas (CenSurE) [21] can perform real-time detection and matching of features taken from images and has been utilised by FrameSLAM in [22]. Bag-of-Words model (BoW) [23] and Vector of Locally Aggregated Descriptors (VLAD) [24] build an image descriptors of fixed lengths by aggregating local feature descriptors around centroids. BoW and VLAD can be used for VPR as shown in [24] and [1], respectively. Histogram-of-Oriented-Gradients (HOG) [25] is a global descriptor used to represent gradient angles, whilst also indicating the gradient magnitude for all image pixels. HOG is computationally efficient and tolerant to appearance changes [26]. Zaffar et al. proposed in [27] a training-free VPR system based on HOG feature descriptor, dubbed CoHOG, that has a low encoding time and good tolerance to lateral shift. As shown in several comparison works [28], [29], VPR descriptors work well with some place changes while not with others. [30] proposes a switching system based on complementary of several VPR techniques [31] to choose the best possible descriptor to deal with a particular place or environmental condition.
The applicability of deep-learning in VPR has been originally studied by Chen et al. in [32] where the authors combined all 21 layers of the Overfeat network [33] trained on ImageNet 2012 dataset with both the spatial and sequential filter of SeqSLAM. In [34], the authors trained two neural network architectures on the Specific PlacEs Dataset (SPED) [35], more specifically AmosNet and HybridNet. Arandjelović et al. [36] introduced a new layer which is based on a generalised VLAD entitled NetVLAD. Khaliq et al. [37] presents a light-weight CNN-based VPR system, with low memory and resource utilization, that is robust to viewpoint and environmental changes. The authors of CALC [38] present a light-weight system that is robust to both viewpoint and illumination variations. Cross-Region-Bow [39] is a VPR technique that searches for regions of interest (ROIs) and uses the convolutional layer to create the representation of these salient regions. Khaliq et al. present RegionVLAD [40], a light-weight CNN-based VPR technique that can filter confusing elements and is able to detect salient features from images. While RegionVLAD is based on the same approach as Cross-Region-BoW, it uses VLAD for feature pooling. In [41], the authors have used the AlexNet ConvNet [42] pre-trained on the ImageNet ILSVRC dataset [43] for object recognition. The authors of [44] and [45] use binary neural networks (BNNs) for VPR. These systems are less computationally demanding than other CNN-based VPR techniques, while achieving similar place matching performance as full-precision systems. However, to run the BNNs, dedicated hardware or an inference engine that enables an efficient computation of bitwise operations is required. Efficiency is the main goal of Arcanjo et al. [46] too, who propose to address VPR by taking inspiration from the drosophila neural system to create a lightweight neural network.
In recent years, considerable effort has been put in creating decentralised VPR architectures. In contrast with centralised architectures where each robot sends the map to a central server that performs place matching computations, in decentralised architectures the visual data gathered from each robot has to be shared between each robotic platform [47]. In [2], the descriptor space of NetVLAD is clustered, and each smaller cluster is sent to a robot to perform efficient decentralised VPR. In [3], a data-efficient decentralised visual SLAM system is presented, where the data association scales linearly with the number of robots present in the Multi-Robot SLAM system. The authors of [4] present a loop detection architecture for performing multi-robot underwater visual SLAM. A common observation in [47], [2], [3] and [4] is that in Multi-Robot SLAM systems, working with a reduced bandwidth can significantly increase the difficulty in transferring the images between multiple autonomous vehicles. This work addresses the data transfer problem using JPEG compression to facilitate VPR applications where the available bandwidth is not capable of transmitting the visual data in an uncompressed form.

III. METHODOLOGY A. Image Compression
JPEG is a compression method that allows the user to select and adjust the amount of compression applied to an image. As JPEG is a lossy compression method, there is a trade-off between image clarity and image size. By applying an increased amount of compression to any given image (e.g. above 90%), artifacts are introduced in the resulting compressed image, which could potentially lead to image alteration as seen in Fig. 1.
The JPEG compression process can be broken down into three main steps. Firstly, the data in a given image is divided into the color and luminance components. As the human perception system is better suited to perceive intensity rather than color information, the latter can be subsampled to reduce the amount of data whilst maintaining the image visually unchanged to the user. Secondly, the data subsampled from the color component is divided into 8x8 pixel blocks. On each block, the Discrete Cosine Transform (DCT) is applied to describe the image content by the coefficients of the spatial frequencies for vertical and horizontal orientations, instead of pixel values. Finally, data quantization is performed, where the higher frequency coefficients are transformed to 0 first. Depending on the amount of compression selected by the user, the subsampling step may be skipped to achieve mild image compression. Conversely, to extremely compress an image, the subsampling step is turned on, and the quantization matrix is selected so that most coefficients are set to 0.
The amount of compression applied to a given image is inversely proportional to the quality of the resulted image. Both quality and compression are parameters of the JPEG compression function, with values in range [0,100]. Thus, the amount of compression applied to the image can be computed as follows: With higher compression, both the image quality and size are reduced, while the resolution of the compressed image remains unchanged.

B. Place matching
Place matching is performed by retrieving the best reference image from the map that corresponds with the visual data taken from a robot's camera, known as the query image. To find the best image in the map, each query image is matched with every reference image in the dataset, using the cosine [39] to generate a similarity score between each query-reference pair: where Q F is the feature descriptor for a query image and R F is the feature descriptor for a reference image. For each query image, a distinct list S of N elements will be created containing the similarity coefficients as follows: where N is the total number of reference images in the dataset and s is in range [0,1]. For each query image, the reference image with the highest similarity score is retrieved as the matching place.
The matching process of a query image descriptor Q F with all the reference feature descriptors R F in the reference map R M can be summarized as follows. In the first instance, the similarity scores are computed using equation (2) between Q F and each R F in R M . Secondly, these scores are stored in a 1D array similar to equation (3). Finally, the reference image that has the highest matching score (from S) is regarded as the matching place for Q F .

C. Performance Metric
VPR performance is evaluated using the percentage of correctly matched images [48], [49]: where N c represents the number of correctly matched query images and N r the total number of reference images in the map. The accuracy has values in range [0,1]. Higher the accuracy, higher the place matching performance of a VPR technique.

IV. EXPERIMENTAL SETUP
This section presents the experimental setup for our work. We present the datasets used together with the VPR techniques employed for testing the effects of compression for VPR.

A. Test Datasets
The test data consists of eight datasets designed for VPR applications. Fig. 4 shows a query-reference pair for each of them. The first dataset used is Campus Loop dataset [38]. It contains 100 query and 100 reference images which pose challenges to any VPR system due to the high amount of viewpoint variation, seasonal variation and also the presence of statically-occluded frames. The second and third datasets are part of Gardens Point (GP) dataset [9] which contains both day and night images, that are divided as follows: 200 query images (day left) and 400 reference images − equally split into day images (day right) and night images (night right). Nordland dataset [50] is the fourth dataset used which captures the drastic visual changes that seasonal variation can have on a place (denoted by the changes in spring, summer, autumn and winter seasons). Since the most notable differences between seasons are seen during the summer and winter seasons, each VPR technique is tested on the summer-to-winter traverses of the Nordland dataset. The fifth dataset used is ESSEX3IN1 [10], which is composed of 420 images, equally split in 210 query and 210 reference images containing perceptual aliasing and confusing places/frames. The sixth dataset is Specific PlacEs Dataset (SPED) [35] which contains 607 query and 607 references images. They are low-quality frames taken from CCVT cameras from around the world under weather, seasonal and illumination variations. SYNTHIA dataset [51] presents a simulated citylike environment, and contains frames in various weather, seasonal and illumination conditions. Oxford Robot Car dataset (ORCD) [52] contains images under illumination and viewpoint changes. Table I presents the size of each dataset (in Megabytes) for different levels of JPEG compression.

B. VPR Techniques
In this work, six well-established VPR techniques are used to show the effects of JPEG compression in VPR scenarios. These techniques are as follows: HOG [25], CALC [38], HybridNet [34], NetVLAD [36], CoHOG [27] and RegionVLAD [37]. All VPR techniques are used as they are presented by their authors with no additional changes being made to neither technique.
As mentioned in section III-A, JPEG compression introduces artifacts while decreasing the quality of the image. As a result, JPEG compression introduces appearance changes in an image, rather than viewpoint changes. The selection of VPR techniques employed in this work can be divided into two main categories: VPR techniques that are robust to viewpoint changes (such as NetVLAD and CoHOG) and techniques that are optimized for appearance changes (such as HybridNet). Thus, we can identify which approach can be easily adapted to deal with extreme JPEG compression rates.

V. RESULTS AND ANALYSIS
In section V-A, we present the effects of JPEG compression on the performance of several VPR techniques.
We discuss the performance of each technique for several levels of compression in terms of accuracy. Furthermore, in section V-B, the details of our JPEG optimized CNN are provided, whilst also presenting a comparison between our model trained on compressed data and other VPR techniques. In section V-C we present the place matching performance of our model on non-uniform JPEG compressed data.

A. Place Matching Performance
By increasing the compression percentage on each dataset, we generally obtain lower results. This can be seen in Fig.  2, where the accuracy (Y-axis) generally decreases with the increase in compression rate. This descending trend in performance is expected due to the fact that an increase in JPEG compression would conclude in a drastic change within the image structure (as observed in Fig. 1).
The results presented in Fig. 2 show that the amount of compression applied to each dataset has a direct effect on the place matching performance. However, each technique is affected differently by image compression. A possible explanation for this decrease in place matching performance can be related to the inability of current VPR techniques to cope with the extreme changes that emerge from including image compression besides the already existing challenges in VPR (viewpoint, illumination, seasonal variations etc.). In particular, we observed that JPEG compression affects more those methods designed to deal with viewpoint changes such as NetVLAD and CoHOG. On the contrary, VPR descriptors that are designed to handle appearance changes present higher tolerance to JPEG compression. The details Seasonal variant datasets such as SPED present a substantial reduction in the place matching performance of NetVLAD, HOG and CoHOG as reported in Fig. 2. It is important to note that HybridNet has been originally trained on the SPED dataset, thus the reduction in performance on this dataset is less severe when compared to the rest of the techniques.
On datasets including illumination variation, such as Gardens Point day-to-night and SYNTHIA, techniques such as RegionVLAD have a moderate decrease in performance throughout the compression spectrum whilst NetVLAD, Co-HOG and HOG lose significant performance on the compressed versions of these datasets. However, the results for SYNTHIA show that it is slightly more stable than Gardens Point. As SYNTHIA is a synthetic dataset, it is less information rich than a real-word dataset such as Gardens Point (refer to Fig. 4). The application of JPEG compression on the SYNTHIA dataset does not alter significantly the image content from the perspective of VPR. This conclusion is supported by Fig. 3 that shows the average entropy [27], [48] (on the Y-axis) in each query dataset resulting from applying different levels of JPEG compression. The reduction in entropy on SYNTHIA is much smaller when compared to other datasets tested. It is worth mentioning that for both Gardens Point day-to-day and day-to-night datasets we use day left images as query images, therefore we only provide the entropy results once in Fig. 3.

1) Model Design:
In an effort to achieve more consistent place matching performance for different levels of JPEG compressed data, we propose a neural-network based VPR technique which is optimized specifically for image compression. The neural network has the same structure as HybridNet, and has been trained on the Places365 dataset [53], which contains approximately 1.8 million images from 365 scene categories. Furthermore, this neural network has been fine-tuned using 97% JPEG compressed versions of the images taken from the above mentioned dataset. We have specifically selected 97% JPEG compression rate as it provides the best trade-off between performance and stability.  The resulting model, entitled 97, achieves great stability and consistent performance in the higher compression spectrum, on a variety of environments and viewing conditions.
2) Model Stability and Performance: We have previously mentioned in section IV-B that each VPR technique has a different performance depending on the type and state of the perceived environment and the quality of the data subtracted from it. To successfully perform VPR tasks in real world applications, it is fundamental to determine the best technique with regards to the above-mentioned environmental variables. Thus, in this sub-section, we present a comparison between the place matching performance of our model and that of other VPR techniques, throughout the entire spectrum of JPEG compression. Fig. 2 shows that JPEG compression has drastic effects on the performance of most VPR techniques, especially when using a significant amount of compression (e.g. 97%). As there is no universal model that achieves the highest VPR performance on all tested datasets, we present in Fig. 5 the accuracy of one of the best performing models, with different amounts of compression applied to each dataset. We have selected the features from the fc6 layer as they achieve the best performance on average.
The performance of our 97 fc6 model is shown in Fig. 5. The VPR accuracy is highly stable across different amounts of JPEG compression. A small decrease in stability can be observed on the SPED dataset, however such a decrease in performance is still less prominent than in any of the other VPR techniques presented. Fig. 6 compares the average VPR performance between our model and the other VPR techniques, across all tested datasets. The results presented in Fig. 6 show that our model has more consistent performance on compressed data and tends to have a steadier decrease in performance throughout the compression spectrum. As previously mentioned, there is a trade-off in the performance and stability of each VPR technique with respect to the amount of image compression applied. While outperformed on low compression ratios by NetVLAD, RegionVLAD and CoHOG, our JPEG optimized model can operate considerably better at the highest compression ratios. This is highlighted in Fig. 2 on the Campus Loop dataset, where NetVLAD achieves the highest place matching performance for compression levels of up to 90%. However, when compression becomes extreme (97%), our model should be used instead as it achieves higher place matching performance as reported in Fig. 5. This observation also applies to the Gardens Point day-to-day dataset. Moreover, on Gardens Point day-to-night dataset, our model outperforms every technique for compression levels of above 80%. However, on the 97% compressed versions of Nordland, Oxford Robot Car and ESSEX3IN1 datasets, our model is outperformed by some VPR techniques presented in Fig. 2. In these cases, the technique that achieves the highest VPR performance should be utilised instead.

C. Non-uniform JPEG Compressed Datasets
In some practical cases, the visual data transmitted by an agent might be subject to bandwidth limitations, raising the need to use highly compressed images. Hence, the query and stored images (e.g. the map) may have different JPEG compression levels applied. For this reason, an analysis on the effects of non-uniform JPEG compression on the performance of our model and other VPR techniques is presented below. Fig. 7 presents the average performance of every VPR technique in scenarios where all datasets are non-uniformly compressed. Apart from being stable on highly JPEG compressed images as discussed in section V-B.2, our model also has consistent performance on datasets where the amount of JPEG compression applied to the query and reference images are in the opposite spectrum. In comparison with HybridNet, our model achieves slightly better overall performance while similar results with RegionVLAD are reported. Fig. 8 presents the detailed results on Campus Loop and SYNTHIA datasets with non-uniform JPEG compression applied. On the Campus Loop dataset, our model achieves the highest VPR performance, outperforming every VPR Fig. 7.
The average place matching performance of our model in comparison with the other VPR techniques presented, in scenarios where the amount of JPEG compression applied to query and reference images greatly differs. technique tested. In section V-A we have shown that SYN-THIA is more stable under JPEG compression (due to its synthetic nature) in contrast to other datasets taken from realworld environments. This observation is also emphasized in Fig. 8, which shows that the performance of most VPR techniques on the SYNTHIA dataset is not drastically affected in the presence on non-uniform JPEG compression, especially when compared with the results presented in Fig. 2.
The results presented in Fig. 7 and Fig. 8 show that our model is more tolerant to non-uniform compression than any of the other VPR techniques tested. Moreover, the results presented throughout this paper emphasize the exceptional performance stability achieved by our neural-network on both uniform and non-uniform JPEG compressed data.

VI. CONCLUSIONS
This paper conducts an in-depth study on the effects of JPEG compression in VPR. We use a selection of wellestablished VPR techniques on a variety of JPEG compressed VPR datasets to present our findings. Our experiments show that techniques which are designed to deal with appearance changes present higher tolerance to JPEG compression in comparison with techniques that are designed to handle viewpoint variations. In an attempt to achieve more stable VPR performance when using JPEG compressed data, we proposed a CNN-based VPR technique that is optimized to handle highly compressed images. The results show that our model is more consistent on both uniform and non-uniform JPEG compressed data than any other VPR technique presented in this work.