{'fraction_non_alphanumeric': 0.022688110281447443, 'fraction_numerical': 0.003938623122999918, 'mean_word_length': 5.1166875784190715, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transferring dense object detection models to event-based data

Event-based image representations are fundamentally different to traditional dense images. This poses a challenge to apply current state-of-the-art models for object detection as they are designed for dense images. In this work we evaluate the YOLO object detection model on event data. To this end we replace dense-convolution layers by either sparse convolutions or asynchronous sparse convolutions which enables direct processing of event-based images and compare the performance and runtime to feeding event-histograms into dense-convolutions. Here, hyper-parameters are shared across all variants to isolate the effect sparse-representation has on detection performance. At this, we show that current sparse-convolution implementations cannot translate their theoretical lower computation requirements into an improved runtime.


INTRODUCTION
Event-based or neuromorphic cameras provide many advantages like high-frequency output, high dynamic-range and a lower power-consumption. However, their sensor output is a sparse, asynchronous image-representation, which is fundamentally different to traditional, dense images.
This hinders the use of convolutional layers, which are an essential building-block of current state-of-the-art image processing networks. Classical convolutions on sparse data, as it is produced e.g. by event-cameras, are inefficient, as a large part of the computed feature-map defaults to zero. Furthermore, sparsity of the data is quickly lost, as the non-zero sites spread rapidly with each convolution. To alleviate this problem, changes to the convolutional layers were proposed.
Sparse convolutional layers [6] compute convolutions only at active (i.e. non-zero) sites. The sub-type of 'valid' or 'submanifold' sparse convolutional layers furthermore tries to preserve the sparsity of the data by only producing output signals at active sites, which makes them highly efficient at the cost of restricting signal propagation. Non-valid sparse convolutions are semantically equivalent with dense convolution layers in that they compute the same result given identical inputs. Valid or submanifold sparse convolution layers, on the other hand, differ from dense convolutions, but still provide a good approximation for full convolutions on sparse data.
Messikommer et al. [8] further introduce asynchronicity into the network. This allows for samples to be fed into the network in parts as they are produced by a sensor, and thus to reduce the latency in real-time applications. Several small batches of events from the same sample can be processed sequentially, producing identical results to synchronous layers once the whole sample has been processed. However, [8] only implemented a proof-of-concept. The project only includes asynchronous submanifold sparse convolutional and batch-norm layers, whereas the sparseconvnet (SCN) project [6] provides a full-fledged library. Furthermore, asynchronous models cannot be trained, as the index_add operation used in the forward function is not supported by PyTorch's automatic gradient tracking. This, however, does not pose a problem, as each layer is functionally equivalent to its SCN counterpart. Therefore, it is possible to train an architecturally identical SCN network and transfer the weights. As the asynchronous property is only relevant during inference, this does not pose a limitation.
An alternative approach is to convert the sparse frame representation to dense frames first, using a learning-based approach [9]. This way, however, one loses all computational advantages that the sparse representation offers. Notably, it is also possible to synthesize events from a dense frame-based representation [3].
Furthermore, a leaky surface layer was proposed by Cannici et al. [1] which integrates the event-to-frame conversion directly into the target network. This way the network becomes stateful, and resembles a spiking model [7].

Contributions and outline
In this work, we use the YOLO v1 model [10] as a simple but powerful dense object recognition baseline. We model sparse networks architecturally identical to YOLO v1 using the SCN [6] and asynet [8] frameworks. These serve as a case study to evaluate the performance of sparse and asynchronous vs dense object detection.
We implement all variants in PyTorch and evaluate the predictive performance and runtime requirements against a dense variant. To this end, we convert the KITTI Vision dataset to events using [3]. This allows us to answer the question if these novel technologies are a viable optimization over dense convolutional layers, or if they fall short of the expectations in practice.
The remaining part of this work is structured as follows: First, section 2 introduces data formats required for the remainder of this work. Next, section 3 details the major changes and additions to the used frameworks. Section 4 evaluates the sparse and dense YOLO versions w.r.t. performance, and section 5 regarding runtime. Section 6 concludes our work by discussing our results and providing an outlook.

DATASET AND FRAME REPRESENTATION
In this work we focus on frame-based object detection models. Dense video-frames capture absolute light intensity of a scene at a specific moment in time. Event cameras, on the other hand, capture discrete changes of brightness at specific locations in space and time as a sequence of events. These are more akin to videos, or image sequences, but without knowledge of the absolute brightness of any location.
Synthetic event sequences can be generated from videos by interpolating to infinite resolution in the time axis and then extracting each change of a pixels value as a discrete event at the (estimated) time of this change. Vice versa, videos can be reconstructed from events, if the absolute information of the starting frame is known. Without such information, one can still try to reconstruct a single image from events, e.g. by assuming the starting point to be an empty, grey image, which might yield good results if the amount of events available is large enough.

KITTI Vision dataset
As we compare sparse with dense convolutional neural networks (CNNs), we require parallel dense image and sparse event data. Therefore, we use the dense KITTI Vision object detection dataset [5] and convert it to events using vid2e [3].
The dataset contains a large collection of urban traffic based scenes captured by cameras mounted on the roof of a car. Each sample is manually annotated with bounding boxes of different object classes like 'Pedestrian', 'Car', 'Cyclist', etc. The intended use is autonomous driving and driverassistance systems.

Optical event data and histograms
Optical event data can be represented in various formats. A simple and lossless encoding are sequences of discrete events storing the spatial and temporal location and the polarity of the change [2], as the amount is usually assumed to be fixed within one dataset.
This format is, however, badly suited to processing with e.g. CNNs, as the sequence length is variable across samples and unbounded. A common format that overcomes this limitation are event-histograms, which accumulate all events into a single frame similar to an image, but showing changes of brightness during the defined interval instead of absolute brightness values at a single point in time.
In this work, we use the event-histogram representation from the asynet [8] framework producing two channels, where each pixel value represents the sum of all observed event changes of negative or positive polarity at this spatial location.

IMPLEMENTATION
In addition to the available dense PyTorch YOLO v1 implementation 1 , we implemented two more networks to be used in our final evaluation: A sparse version of YOLO v1 implemented in the SCN framework and an asynchronous sparse version implemented in the asynet framework. Training and evaluation of all networks is performed using the asynet framework to ensure comparability of results. For reproducibility, we make our implementation available as open-source on github 2 .
The sparse models follow the YOLO v1 architecture, but use specialized sparse or asynchronous sparse layers in the convolutional block, followed by standard PyTorch linear layers. In the SCN sparse model, the convolutional block is followed by a sparse-to-dense layer that converts the sparse tensor into a dense representation for further processing. In the asynet asynchronous model such a layer is not necessary, as the model does not support training anyway and the dense feature map tensor is passed through the network alongside the sparse events as part of the sparse representation.
Both sparse models employ submanifold sparse convolution layers where the dense network uses convolutions with stride 1 to achieve maximum performance. We adapted the trainers for dense and sparse object detection models already implemented in the asynet code for improved logging and debugging and added early stopping. However, neither the SCN, nor the asynet framework contained sufficient functionality to directly implement a YOLO v1 network.

Sparseconvnet extensions
In the case of SCN, the deficit was minimal, as it only lacked the 'same'-padding feature in its sparse convolutional layer. To get around that limitation, we chose a rather inefficient but easy way of converting a sparse tensor into a standard dense PyTorch tensor, pad this dense representation, and then convert it back into a SCN sparse tensor. This does not affect our evaluation, as it can be easily excluded from the runtime evaluation carried out via profiling, and does not change the results of the layer computations.

Asynet extensions
The asynet framework, however, was missing a layer type. The existing 'asyncSparseConvolution2D' layer implements an asynchronous valid or submanifold sparse convolution only. The project does not contain an implementation of an asynchronous (non-valid) sparse convolution. We therefore implemented the asynNonValidSparseConvolution2D layer, based off the asyncSparseConvolution2D implementation. To ensure correctness, we again specified test cases to verify our implementation.
Additionally, the original code did not filter duplicate events within a sequence, causing each active site to be processed as often as the number of duplicate events (at the same spatial location) in the sequence. This behaviour caused the runtime to increase by several orders of magnitude, while also producing incorrect results in case of duplicate events.

Dataloader
We implemented a dataloader for the KITTI Vision dataset analogously to the already available dataloaders for various other datasets (NCaltech101 among others). We adapted code available through the dataset's release site [4] for converting the ground truth bounding boxes and labels into the commonly used format defined by the Pascal VOC dataset. Each sample is converted to events at runtime because of the enormous storage overhead of preprocessing the whole dataset. The spatial locations of the events are then rescaled to the required tensor size, and finally accumulated into an event histogram. Additionally, we implemented a version without the conversion into events to be used to train the dense YOLO network on the original images.

ERROR ANALYSIS
The intuition behind sparse CNNs is to speed up, and reduce energy consumption of, dense CNNs by eliminating unnecessary computations. As such, we require sparse CNNs to match the prediction performance of dense CNNs, while reducing resource consumption.
We start by verifying the first condition, namely recognition performance matching that of the dense model. performed only with limited redundancy, as can be seen in table 1. As the goal of this evaluation is a qualitative comparison of different architectures, rather than trying to achieve state-of-the-art results, it is acceptable to omit hyper-parameter tuning and use the same parameters for all models.
A proper convergence of each training run, as well as the absence of strong outliers within the performed experiments, minimizes the risk of non-representative and non-reproducible results.

Sparse YOLO vs dense YOLO
We first compared the dense YOLO network trained on dense images directly with our structurally identical sparse implementation trained on 42ms event-windows. The mAP score shows the sparse model to perform approximately 20 % better than the dense baseline. This indicates that both the conversion from dense images to sparse events and our model implementation work as intended.
The increase in performance can be explained by the availability of additional information: The dense model is restricted to a single image per sample, the sparse model, however, is trained on events synthesized from a sequence of images. While events encode only the change and lose the information about absolute brightness, it can be argued that the change, which effectively encodes moving edges, is more beneficial to object recognition than colour and absolute brightness.
The maximum achieved mAP of about 23% is significantly lower compared to values YOLO v1 reportedly achieved on other datasets. This is likely due to differences of the mAP calculation 3 . However, we use the same calculation throughout this work, which makes our results comparable to each other.
The 'YOLO loss', as presented in the original YOLO paper [10], shows considerably larger differences between the sparse and dense variants. This metric, however, constitutes a loss function to be optimized during training and is not necessarily suited to compare the performance of different models.

Sparse vs dense convolutions
Training the dense YOLO network on event histograms instead of dense images did not improve its performance notably. It consequently also performed approximately 15 % worse than the sparse model trained on the same data. This indicates the submanifold sparse convolutions, which are the only semantic difference from the dense model, essentially contribute to the ability of the model to process sparse event data.
While similar results might be achieved using only dense convolutions by altering the model structure, it is stunning that submanifold sparse convolutions enable the transfer of a model optimized for dense images to events without requiring additional hyper-parameter tuning.

Event-window size
Against our expectations, event-window size within a reasonable range has little to no effect on the performance. We tested several configurations, starting from 42 ms, which covers exactly one frame of the original dense dataset captured at a frame rate of 24fps. As this frame rate is, however, rather low, and 42 ms contains a huge amount of events, we tested mainly values below that.
As each training run takes more than one day on our available hardware, we decided to use fixed, pre-trained weights for initialization instead of averaging over multiple, randomly initialized runs to exclude effects of weight initialization from this evaluation. All sparse test cases were trained from the same weights, pre-trained for 121 epochs using a window size of 33 ms. There is no noticeable qualitative difference in the evaluation loss and mAP scores between the tested window sizes of 8 ms, 16 ms, 25 ms, 33 ms, 42 ms and 50 ms.

RUNTIME ANALYSIS
Sparse CNNs claim to be more efficient in terms of number of operations and subsequently runtime and energy consumption. To verify this claim we profiled the three different implementations using cProfile 4 .
For these experiments we chose to evaluate the full YOLO v1 network on real data. This ensures a realistic ratio of active sites over locations in the tensor, as well as number of events per active site. While in the dense framework all convolution layers are essentially the same layer type, all unstrided sparse convolutions can be translated to more efficient submanifold sparse convolutions, while strided sparse convolutions have to be implemented as non-valid sparse convolution layers.
Due to the high runtime of the asynchronous framework, all models are profiled over 1042 fixed samples (14 %) of the KITTI Vision dataset.

Dense vs sparse
Although in theory more efficient on sparse data than dense convolutions, sparse CNNs still show higher runtimes. Due to highly optimized code and better hardware support for the massively vectorized operations of the standard dense convolutions implemented in the PyTorch framework, the research oriented proof-of-concept implementations of sparse convolutions, while also highly optimized in the SCN framework, cannot compete in real use-cases yet. Figure 2a shows the cumulative runtime of those layers that are not identical in dense and sparse networks. Convolutions in sparse networks are split into (non-valid) sparse-convolutions and submanifold-sparse-convolutions. Here the performance difference is most significant, with an increase in runtime of more than three times. Furthermore, convolutional layers usually constitute the largest part of CNNs.
Batch norm layers only experience a small loss in performance. I/O-Layers, which set up additional data structures to be passed through the network with each sample to enable efficient computation of the convolutional layers, are only needed in sparse networks and provide an additional notable overhead. As the YOLO v1 model contains two strided convolutional layers and (b) Asynchronous network shows very high runtime (approx. three orders of magnitude higher than the synchronous implementation). Y-axis in logarithmic scale due to extreme differences. only one actual input layer, however, about two third of this overhead can be attributed to our implementation of the 'same'-padding option as described in section 3.1.

Batch size
Repeating this experiment with a smaller batch size of 1 (instead of 30 in the previous evaluation) revealed sparse layers don't suffer as much overhead from smaller batch sizes as dense layers, or rather, in the reverse direction, don't gain as much from predicting more samples at the same time using higher batch sizes.
For convolutions, the gap between dense and sparse layers closes significantly, while sparse batch norm layers actually overtake their dense counterpart. For sparse networks, I/O layers show a similar overhead to convolutions.

Synchronous vs asynchronous
The results of profiling the asynchronous sparse CNN implemented in the asynet framework are far from encouraging. Due to the experimental and little-optimized implementation, the asynchronous convolution layers show an increase in runtime of roughly three orders of magnitude, as seen in Figure 2b. While the synchronous framework essentially works on event histograms accumulating all events at the same location, for the asynchronous framework multiple events per active site are an important feature of the data. An asynchronous network processes a sample in multiple sequences representing discrete time steps. An active site is only processed in any distinct sequence if there exists an event at that active site at a time that falls into the range of that sequence.

Asynchronous sequence count
To make use of the asynchronous nature of the network, a sample will usually be split into a number of event-sequences, which are then processed in series. When examining the effect of the number of theses sequences, we would expect an increase in runtime for larger sequence counts due to possibly duplicated active sites, bounded by the runtime of the 1-sequence-baseline   times the number of sequences. For small numbers of sequences the number of active sites in each sequence will not decrease notably, as there usually is more than one event at most active sites in a sample, thus performing close to this upper bound. For large sequence counts, however, we expect a sub-linear increase in runtime due to a decreasing number of active sites per sequence. While profiling the model using two sequences exactly matches our expectations, as shown in Figure 4, the model exceeded the upper bound for three sequences.
Further analysis showed that the unexpected increase in runtime is caused in low-level functions like tensor formatting. We believe this to result from overhead due to extremely high memory utilization. While profiling with three sequences required 360 GB of RAM, profiling of higher sequence counts exceeded our available resources and was thus omitted. Therefore, confirmation of our claim of sub-linear increase in runtime for high sequence counts is left for future work.

Theoretical runtime
Given the data is sufficiently sparse, the sparse convolution based method should have a significant advantage.
• The dense convolution convolves the filter with every position of the input tensor, yielding as many convolution operations as there are unique locations in the tensor. • The sparse convolution convolves the filter with every active site, with the active sites being a subset of the unique locations in the tensor.
Therefore, the complexity of the sparse convolution is generally lower than that of the dense convolution. Additionally, submanifold sparse convolutions further reduce complexity by only computing those parts of the convolution at each active site where filter and active sites overlap.
The main gain of submanifold sparse convolutions, however, lies in preventing the increase in the number of active sites, additionally reducing the complexity for the following layers. In practice, sparse convolution layers require the construction of a so-called rulebook for each sample to efficiently compute the necessary convolutions, as detailed in [6]. While this creates some overhead, it is outweighed by the gains of only processing the active sites. Furthermore, blocks of consecutive submanifold convolution layers pass through the rulebook and ensure it stays valid, so that it only needs to be computed once, further reducing the complexity.
Asynchronous-sparse-convolutions split the events into multiple sequences, and within each sequence behave like synchronous sparse convolutions. Each active site of a synchronous-sparseconvolution is caused by at least one event, but might accumulate many events that happened at the same spatial location. Therefore, each active site is processed in at least one sequence, but at worst case in all of them. Such layers thus have at least as high a complexity as their synchronous counterparts, but stay within a predictable margin.
However, the sparse nature of the data hinders SIMD processing and the use of on-chip cachestwo techniques that are crucial for reaching high performance on current hardware.

CONCLUSION
In this work we have evaluated the prediction performance and runtime of sparse and asynchronoussparse CNNs with respect to classical dense CNNs. Our experiments have shown that sparse CNNs can match the performance of their dense counterparts without requiring additional hyperparameter tuning.
The approach works well with synthetically generated events from an existing dense dataset, which we believe will be beneficial for the adoption of this technology. Whereas the production of new high-quality datasets for specialised application domains can be very expensive, dense datasets are quite abundant in comparison, even with the constraint of requiring image-sequences to be applicable for conversion to events.
We think that asynchronous-sparse CNNs are a promising new concept that may find use in real-time applications due to the extremely low latency. In practice, however, these concepts are not yet sufficiently optimized.
We have extended the experimental asynet framework for asynchronous-sparse CNNs and shown that sparse CNNs match classical dense CNNs in prediction performance. On the other hand, we found that the runtime performance of the evaluated frameworks cannot yet match dense networks, and especially the asynchronous framework can at this point only be seen as a proof-of-concept. The ease of transfer from dense to sparse networks and the potential gains in runtime will hopefully incite further research into these promising technologies. We believe the evaluated sparse CNN frameworks to be limited by code inefficiencies and lacking hardware support, but in theory to be a viable optimization of CNNs.