{'fraction_non_alphanumeric': 0.03639438525283274, 'fraction_numerical': 0.008320649416539828, 'mean_word_length': 4.630546562559513, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Event-VPR: End-to-End Weakly Supervised Network Architecture for Event-based Visual Place Recognition

Traditional visual place recognition (VPR) methods generally use frame-based cameras, which is easy to fail due to dramatic illumination changes or fast motions. In this paper, we propose an end-to-end visual place recognition network for event cameras, which can achieve good place recognition performance in challenging environments. The key idea of the proposed algorithm is firstly to characterize the event streams with the EST voxel grid, then extract features using a convolution network, and finally aggregate features using an improved VLAD network to realize end-to-end visual place recognition using event streams. To verify the effectiveness of the proposed algorithm, we compare the proposed method with classical VPR methods on the event-based driving datasets (MVSEC, DDD17) and the synthetic datasets (Oxford RobotCar). Experimental results show that the proposed method can achieve much better performance in challenging scenarios. To our knowledge, this is the first end-to-end event-based VPR method. The accompanying source code is available at https://github.com/kongdelei/Event-VPR.


I. INTRODUCTION
Visual place recognition (VPR) [1] is a very challenging problem in the field of computer vision and mobile robots. In computer vision, VPR can be used to retrieve visual information or cross-time place information in a large-scale image database with geographic information annotation, or be used in interactive 3D vision applications such as augmented reality (AR). In mobile robots, the ability of robots to recognize visual places in GPS-denied environment is one of the key capabilities for autonomous localization and navigation. In simultaneous localization and mapping (SLAM), VPR is an important component of loop closure detection [2] [3], which can be used to detect candidate loop-closures and eliminate accumulated errors through global optimization for globally consistent pose estimating and mapping.
At present, there are many solutions to the VPR problem in large-scale environments. For those existing solutions, monocular, binocular, panorama camera and other framebased vision sensors are widely used. However, since framebased vision sensors suffer from issues such as illumination changes, motion blur and redundant information, which makes traditional VPR methods difficult to deal with recognition tasks in some challenging environments. Besides, most existing methods are based on the appearance of scenes [4]. Due to various reasons, such as day-and-night, weather and seasonal changes, the appearance of the same place will change greatly at different time. In addition, the appearance of some places at long distances may be very similar. These issues pose great challenges to the existing large-scale framebased VPR methods.
In contrast to traditional frame-based VPR methods, we propose a novel VPR method using event cameras in this paper. Event cameras are neuromorphic visual sensors inspired by the biological retina, and work in a completely different way from frame-based cameras: they use addressevent representation (AER) and output pixel-level brightness changes (called events) with microsecond resolution to generate sparse asynchronous event streams. Event cameras have the advantages of low latency, high temporal resolution, low bandwidth, low power consumption and high dynamic range [5], which can effectively overcome the problems existing in typical frame-based cameras. To achieve robust VPR using event cameras, we propose a novel end-to-end visual place recognition network for event cameras (Event-VPR). The key idea, as shown in Fig. 1, is to apply NetVLAD to EST voxel grid representation generated by event streams. To the best of our knowledge, this is the first end-to-end VPR method for event cameras. Experimental results on multiple datasets with different weather and scenes show that the proposed method is superior to the traditional framebased VPR methods, and can effectively solve the challenges of large-scale scenes, high dynamic range and long-term adaptability in visual place recognition.
The main contributions of this paper are as follows: • A novel end-to-end weakly supervised network pipeline of visual place recognition for event cameras is proposed, which directly uses event streams as input. • Different event representations, network structures and parameters of the proposed network are compared to show how it affects the total performance. • The performance of the proposed method against framebased methods on different datasets with different weather and seasons is thoroughly evaluated and the source code of our method is open-sourced 1 .

II. RELATED WORK
Visual sensors are the main sensor types for place recognition due to their low cost, low power consumption and abundant scene information. Nowadays, most popular VPR methods generally use frame-based visual sensors and appearance-based [4] approaches to realize large-scale place recognition. In this case, the VPR problem can generally be transformed into a large-scale geo-tagged image retrieval problem, and that can be solved by image matching. Extensive researches on how to represent and match images have been carried out [1] [6]. Those methods usually use traditional feature extraction techniques (such as SIFT [7] and ORB [8]), and local descriptor aggregation techniques (such as BoW [2] [3] and VLAD [9] [10]) to establish a higher-order statistical model of image features. With the rise of deep learning, some works begin to use off-theshelf convolutional networks (such as Overfeat [11], VggNet and AlexNet [12]) as trainable feature extractors. Some researchers also revised VLAD into a trainable pooling layer (such as NetVLAD [13] and DenseVLAD [14]) to obtain image descriptor vectors as compact image representations. In the retrieval and matching process, sequence-based matching is a widely recognized method. The most known work is SeqSLAM [15], which searches highly similar short image sequences for VPR. Recently, researchers also tried to further improve the recognition performance from different aspects. For example, most structure-based methods use structural information such as repeated edges and semi-dense maps of the scene [4], [16], [17] for place recognition. In recent years, there are also some semantic-based works for VPR that mainly use semantic information such as landmarks, texts and objects in the scene. [18]- [20].
Though traditional frame-based VPR methods have developed rapidly over the past decade, they are still difficult to solve some problems in challenging scenarios (such as illumination changes and motion blur) due to the inherent defects of frame-based cameras. Compared to standard framebased cameras, event cameras have many advantages such as high dynamic range, high temporal resolution and low latency [5]. Due to the above advantages, event cameras have drawn more and more attention. However, to the best of our knowledge, there are still few works related to the event-based VPR problem. Among them, Milford et al. first 1 https://github.com/kongdelei/Event-VPR tried to migrate SeqSLAM to the event camera in 2018 and completed a relatively basic recognition experiment based on event frames [21]. Recently, they proposed an event-based VPR scheme with ensembles of spatio-temporal windows (Ensemble-Event-VPR) [22]. This method uses event bins with different numbers of events and sizes of temporal windows to reconstruct a group of intensity frame sequences using E2Vid [23] for place recognition. However, this method is not a direct event-based method but needs to convert the events into intensity frames, which is still a frame-based VPR method in essence. Different from that, in this paper, we propose a novel end-to-end event-based visual place recognition network (Event-VPR) that directly uses event streams, which achieves excellent recognition results even in challenging environments.

III. METHODOLOGY
In this section, we will describe the network architecture of Event-VPR in detail, including the various module components of our algorithm and the main steps of network training.

A. Problem and Pipeline
Problem Definition. Define a database of events D = {P, E} that contains n geo-location coordinated P = {P 1 , · · · , P n } under a fixed reference system and n groups of event bins E = {E 1 , · · · , E n }, where each place coordinate P i corresponds to several event bins E i = {E i1 , · · · , E im } and each event bin E i j is collected by using an event camera in the area near the place coordinate P i . The area of coverage (AOC) of all sub-areas is approximately the same, i.e. AOC (P 1 ) ≈ · · · ≈ AOC (P n ). The problem of event-based VPR can be formally defined as follows: Given a query event bin denoted as E q , our goal is to retrieve several event bins from the database D that are most similar to E q , thus obtaining their corresponding geographic location coordinates P q . To this end, we design a deep neural network to learn a function f Event-VPR (·), which is used to map a given event bin E q to a global descriptor vector is a distance function (such as euclidean distance function). Our problem then is simplified to find the place coordinates P q of the sub-area such that the global descriptor vector v * from one of its event bin gives the minimum distance with the global descriptor vector v q from the query, i.
Our Pipeline. The event-based visual place recognition (Event-VPR) proposed in this paper is a novel end-to-end VPR method using event-based camera. The key idea of the algorithm is as follows: Firstly, we divide consecutive event stream into event bins and convert event bins into EST voxel grid representations using MLP-based kernel. Then, a cropped ResNet34 network [24] is used to extract visual features of EST voxel grids. Next, a VLAD-based local aggregated description layer is used for feature descriptor aggregation. Finally, the triplet ranking loss is used to train the network with weakly supervised training. Corresponding to the aforementioned key idea, our pipeline is mainly divided into the following 4 parts: EST voxel grid representation, feature extraction convolution network, feature aggregated description layer and triplet ranking loss. Its architecture is shown in Fig. 2.

B. Events and EST Voxel Grid
Event-based Data. The pixel array of the event camera is capable of independently and logarithmically responding to pixel-level brightness changes (i.e. L . = log(I), where I is photocurrent) and triggering sparse asynchronous events E = e 1 , · · · , e N | e k ∈ R 4 . To be exact, without considering fixed pattern noise (FPN), the brightness change at the pixel (x k , y k ) T at time t k is given by: where ∆t k is the time interval between last triggered event and current triggered event of a pixel. When the brightness change of a pixel reaches the contrast threshold ϑ (here ϑ > 0), the pixel triggers an event e k = (x k , y k ,t k , p k ) T . Here, p k is event polarity which is given by: In real sensor, positive events (ON) and negative events (OFF) can be triggered according to different contrast thresholds ϑ . Furthermore, the events can be summarized as an event measurement field with polarity defined on the 3D continuous spatio-temporal manifold: where δ (·) denotes the dirac pulse defined in the event domain and is used to replace each event.
EST Voxel Grid. The output of an event camera is the consecutive event stream. In order to use deep neural networks to extract features from event stream, we need to convert the event streams into a representation that convolutional network could process. Nowadays, typical representation methods of events roughly include motion-compensated event frame (EF) [25], 4-channel event count and lasttimestamp image (4CH) [26], and event spike tensor (EST) [27], etc. In addition, the events can also be converted into traditional frame-based video (e.g. E2Vid) [23]. In this paper, we firstly divide event stream into event bins, then we use the voxel grid of event spike tensor (EST) to represent event bins. In order to obtain most meaningful visual feature information from the event measurement field, we convolve it with a trilinear voting kernel k(x, y,t). Therefore, the convolution signal becomes: After the kernel convolution, the signal (4) can be periodically sampled on the spatio-temporal coordinates: Here, (W, H) is the pixel size, t 0 is the start timestamp, ∆t is the size of time blocks, C is the number of time blocks (that is, the number of channels). We replace the manually designed kernel in (5) with a multi-layer perceptron (MLP) to generate EST voxel grid as an end-to-end events representation, as shown in Fig.  3, where C is the number of channels, B is the batch size. The MLP receives the normalized timestamp of the event as input and has 2 hidden layers, each with 30 neurons. The value generated by MLP for each event is put into the corresponding voxel grid coordinates. EST voxel grid can be optimized according to specific tasks to optimize the performance of the whole network.

C. Feature Extraction and Aggregation
Feature Extraction Network. To extract features from the EST voxel grid, we use the ResNet34 pre-trained in eventbased hand-written number recognition tasks [24]. In order to migrate it to our VPR task, the original network needs to be cropped. We modify the number of input channels of the first convolution layer of ResNet34 to make it suitable for EST voxel grid as network input. Meanwhile, in order to connect the feature description aggregation layers, we remove the last average pooling layer and the full connection layer at the end of the ResNet34. The network output x = f ResNet34 (V EST ) is the feature maps of the EST voxel grid. Here, x is the w × h × D-dimensional feature tensor obtained by the feature extraction network. For ResNet34, w × h = 7 × 7.
Feature Aggregation Network. After getting features for the EST voxel grid, we need to aggregate the features for descriptor matching. We use vector of locally aggregated descriptor (VLAD) [9] [10] which is a trainable descriptor pooling method commonly used in the fields of image retrieval. As shown in Fig. 4, we interpret the w × h × D-dimensional feature maps x output by the feature extraction network as M D-dimensional local descriptors x 1 , · · · , x M | x i ∈ R D as input, and K D-dimensional cluster centers c 1 , · · · , c K | c k ∈ R D as VLAD parameters. The normalized output of descriptor vector V VLAD is a D × Kdimensional matrix, which is given by: where (x i − c k ) is the residual vector of descriptor x i to cluster center c k , andā k (x i ) denotes the soft assignment of descriptor x i to cluster center c k , which is given by: It assigns the weight of descriptor x i to cluster center c k according to their proximity distance. Where, w k = 2αc k , b k = −α c k 2 , α > 0, andā k (x i ) ∈ [0, 1]. According to (7), soft assignment can be decomposed into a convolution layer and a soft-max layer, and the weight {w k } and bias {b k } of the convolution layer are taken as independent trainable parameters together with the clustering center {c k }. Finally, the aggregated vector needs to be intra-normalized and L2normalized to produce the final global descriptor vector v ∈ R Ω , v 2 = 1, Ω = D × K for event bins that can be used for efficient retrieval.

D. Network Training
Training Triplet Building. We use the metric learning to train Event-VPR in an end-to-end manner to learn the function f Event-VPR (·) that maps the input of event bins E into global descriptor vectors v ∈ R Ω . In the process of network training, the training query E q and its geographical location P q is given. It is necessary to select suitable best positive and hard negatives from the database D = {P, E}. In order to improve the search efficiency, firstly, all samples within  Fig. 4. Overview of VLAD-based Pooling Layer. Residual vector and soft assignment are statistics from descriptors to clusters, and the convolution layer and clusters are learnable through end-to-end training.
the range of geographical distance λ are selected as potential positives E pos,i according to the location P q of the query: Then, among these potential positives, the positive with the smallest descriptor vector distance is selected as the best positive, and its descriptor vector is: When selecting negatives, firstly, all samples outside the range of geographical distance δ are selected according to the location P q of the query, and then n sample of them are selected as randomly sampled negatives E neg,i : Then, among these randomly sampled negatives, the samples that violates the margin condition are selected as the candidate hard negatives E hard-neg,i : where m is a constant parameter, representing the margin between d γ (v q , v pos,i ) and d γ (v q , v neg, j ). In order to improve the training efficiency, we select the n neg samples with the smallest descriptor vector distance as the hard negatives for training, where n neg ∈ [0, N neg ], n neg n sanple .
Loss Function. Based on the above method, we use the data from the event camera datasets to obtain a set of training tuples from the training set, where each triple is represented as ξ = E q , E best-pos , E hard-neg, j . Here, E q is the query, E best-pos is the best positive, and E hard-neg, j is a group of hard negatives. If their global descriptor vector is γ = v q , v best-pos , v hard-neg, j , the descriptor vector distances of query E q to the best positive E best-pos and the hard negatives E hard-neg, j are defined as d γ v q , v best-pos and d γ v q , v hard-neg, j respectively. The loss function is designed to minimize the global descriptor distance between the query and the best positive, and to maximize the distance between the query and the hard negatives. We use weakly supervised triplet ranking loss which is defined as follows: where m denotes the margin, and [·] + = max(·, 0) means the loss takes a positive number, i.e. L triplet (γ) ≥ 0.
IV. EXPERIMENTS In this section, we carried out experiments on multiple datasets, including the MVSEC [28], DDD17 [29] and Oxford RobotCar datasets [30] to verify the effectiveness of the proposed method through quantitative and qualitative experimental results (A detailed description of the experimental configuration is given in APPENDIX). We conducted four experiments to evaluate our proposed method. Firstly, we evaluated the performance of the Event-VPR in different driving scenes, and verified its long-term robustness. Then, we compared the Event-VPR with frame-based VPR algorithms on the same dataset, and provided numerical analysis to show the performance of Event-VPR. Finally, we performed network ablation study on Event-VPR from two aspects (event representation and feature extraction network), and the experimental results proved the advantages of each module in our method.

A. Performance Comparison in Different Scenarios
In the first experiment, we validated the performance of our Event-VPR in different scenarios of MVSEC, DDD17 and Oxford RobotCar datasets. We selected several challenging sequences for our experiments. For example, the night scene sequence in the MVSEC dataset is dim with little light. There are many similar scenes and high dynamic range scenes (e.g. clouds and tunnels) in the expressway scenes in the DDD17 dataset. As shown in Fig. 5, there are also changing weather and seasonal conditions in the Oxford RobotCar dataset. For instance, there are light reflection from the surface of water on rainy day and the bright street lamps at night, which result in a high dynamic range scenario for visual cameras.  Table I, the experimental results on the MVSEC dataset show the recognition performance (Recall@N) of the method in daytime and nighttime scenes. On the MVSEC dataset, we trained 2 daytime and 3 nighttime sequences together, and then tested each sequence separately. The results show that the Recall@1 of our model in night sequences can achieve 97.05% on average, which is almost the same as that in the daytime sequences (Please refer to APPENDIX to see how we select training and test sets). Results on Oxford RobotCar. Fig. 6 shows the recognition performance of our method under various weather and seasons on the Oxford RobotCar dataset. The experimental results of model trained on all sequences and tested on different weather sequences are shown in Fig. 6(a) (Please refer to APPENDIX for details). Our model can achieve high performance in different weather conditions. We trained models in day and night sequences separately, and tested on untrained scene sequences (e.g. overcast, rain and snow).

Results on MVSEC. As shown in
The experimental results of model trained on all daytime sequences (including sun and cloud) but tested on different weather sequences are shown in Fig. 6(b). We noted that the test results on overcast and snow sequences are almost the same as those on sun and cloud sequences, and the difference between Recall@1 is only about 2.15%. And the test results in rain and night sequences are poor, with Recall@1 being 14.23% lower on average. The experimental results of model trained on all night sequences and tested on night and day sequences respectively is shown in Fig. 6(c). Due to the large noise of the events in the night sequences, the trained model has poor performance in the daytime scenes.
Results on DDD17. In addition, in order to verify the generalization performance of our method, we use the network model trained on the synthesis event streams of Oxford RobotCar dataset, and then we test the trained model on freeway sequences of DDD17 dataset recorded by real DAVIS camera (Please refer to APPENDIX for details). Surprisingly, the Recall@5 of our method on the night freeway sequences in DDD17 dataset is up to 92.2% without any transfer learning or fine-tuning of the trained network. It means that just using synthesis events for model training can obtain excellent results on events recorded by real event camera. All sequences results are shown as Table II.

B. Performance Comparison with Different VPR Methods
In order to compare the performance of our Event-VPR with conventional frame-based method, we compared Event-VPR with NetVLAD [13] and SeqSLAM [15] on MVSEC and Oxford RobotCar datasets, and the results are shown in Table III. For fair comparison, we adjusted the spatial All-all All-cloud All-night All-overcast All-rain All-snow resolution of the image frames to be the same as that of the event bins. The results show that our method is better than NetVLAD and much better than SeqSLAM on MVSEC dataset, since most images recorded by DAVIS camera are dim and Event-VPR has advantages under this condition. On Oxford RobotCar dataset, the recall@1 of Event-VPR is about 26.02% higher than SeqSLAM but about 7.86% lower than NetVLAD. We think the reason is that a lot of fixed pattern noise is added to the event streams manually generated by V2E [31]. Considering that the event cameras are not mature enough (e.g. low spatial resolution, high noise level, etc.), therefore Event-VPR shows similar performance to frame-based method.

C. Impact of Different Event Representations
In this experiment, we explored the Event-VPR performance with different event representations. In addition to the EST voxel grid described in our method, we have also tried several other event representations, including event frame (EF), unipolar event voxel grid (EVG), and 4-channel event count and last-timestamp image (4CH). We all used ResNet34 as the feature extraction network in this experiment. As shown in Fig. 7, since EF discards timestamps and EVG discards event polarity, EST voxel grid has great advantages over other representations. Due to 4CH discards earlier timestamps, it is also not as good as EST voxel grid. EST voxel grid not only retains the local spatio-temporal neighborhood information of events, but also extracts effective features through using learnable kernel to suppress the noise of events.

D. Impact of Different Network Structures
In this experiment, we explored the Event-VPR performance of different network structures. In addition to the ResNet34 described in our method, we also trained VGG- 16 in the whole experiment. As shown in Fig. 8, experimental results show that different feature extraction networks have no significant impact on the network performance, while a larger network increases the computing resources without improving the accuracy. We guess that a larger network needs more parameters, which makes the network easier to overfitting. We finally choose ResNet34 as the feature extraction network of our pipeline. However, we can replace it with other different convolutional networks according to our own needs and computing resources.

V. CONCLUSIONS
We proposed an end-to-end method (Event-VPR) to solve the problem of large-scale place recognition by using event cameras. We showed that our Event-VPR method based on events is more robust to changes in the surrounding environment caused by weather and time changes than conventional frame-based VPR methods. Though we get very good results, it should be noted that the spatial resolution of event cameras is still very low, so there are still some deficiencies compared to frame-based VPR methods. In the future, we will try to combine standard cameras and event cameras to implement a hybrid network architecture for visual place recognition based on frames and events.

A. Details on Network Training
Caching. In the process of network training, it is necessary to retrieve the descriptor vector of samples to calculate the descriptor distance. In order to improve the training efficiency, we build a cache for the descriptor vectors of the entire database and use the cached descriptor vectors to select the best positive and hard negatives. Because the model parameters will be continuously adjusted during the training process and the descriptor vectors of the network output will also be continuously changed, we will update the cache regularly during the network training. Recalculating the cache every 500-1000 training queries can achieve a good balance between the duration of the training period, the convergence speed of the network and the quality of the model.
Training Data Loading. Different from traditional image frames, sparse event bins have the problem that the amount of data is not fixed in the process of loading data, which makes it difficult to distinguish each sample during batch training. When we build a batch process, we add a column of indexes to each event bins, arrange the query, the best positive and the hard negatives in turn, merge them into a batch process and send them to the network for training. Then the descriptor vectors are divided according to the indexes.

B. Dataset Configuration
In our experiments, we use MVSEC [28], DDD17 [29] and Oxford RobotCar datasets [30]. The MVSEC and DDD17 datasets are event-based driving datasets recorded in the real environment. We select intensity images and event bins of 5 outdoor driving sequences from MVSEC dataset (including day-and-night scenes, using left-eye DAVIS camera) and 6 outdoor driving sequences from DDD17 dataset (including urban and freeway scenes), which include dramatic illumination changes and scene structure changes. In addition, Oxford RobotCar dataset is a standard dataset commonly used in VPR. We use the event synthesizer V2E [31] to convert the image sequences recorded by the center camera of the three-eye vision sensor (Bumblebee XB3) in Oxford RobotCar dataset into corresponding event streams. We tried our best to select sequences with the same trajectory under different weather conditions, including sun, cloud, rain, snow and night scenes. We think that such scenarios are more challenging for testing the performance of VPR methods.
We randomly divide the sequences of the same trajectory into geographically non-overlapping training and test sets (detailed results are shown in Table IV V VI VII VIII). In the MVSEC dataset, we select about 40k training samples and 10k test samples from 5 sequences. In the DDD17 dataset, we select about 120k test samples from 6 sequences. In Oxford RobotCar dataset, we also select about 50k training samples and 12k test samples from 11 sequences. For parameters, we choose potential positive distance threshold λ = 10m, potential negative distance threshold δ = 25m, and true positive geographical distance threshold ϕ = 20m in MVSEC and Oxford Robocar datasets. Moreover, all parameters in DDD17 dataset are the same except that the true positive geographical distance threshold is ϕ = 60m.
Generally, the performance of VPR methods is evaluated by Recall. In this paper, we use Recall@N to evaluate the performance of our algorithm. Specifically, for each query, the N-nearest neighbor samples are retrieved as positive samples according to the descriptor distance. If at least one of samples is less than ϕ away from the query according to their geographical distance, it is considered to be correctly identified. Then, we calculate the Recall@N with different N. For the query set, we traverse descriptor vectors of all queries to calculate Recall@N which is the percentage of correctly identified query descriptors. In our experiments, we restrict our analysis to N = {1, 5, 10, 20}.