ESS: Learning Event-based Semantic Segmentation from Still Images
  Retrieving accurate semantic information in challenging high dynamic range
(HDR) and high-speed conditions remains an open challenge for image-based
algorithms due to severe image degradations. Event cameras promise to address
these challenges since they feature a much higher dynamic range and are
resilient to motion blur. Nonetheless, semantic segmentation with event cameras
is still in its infancy which is chiefly due to the novelty of the sensor, and
the lack of high-quality, labeled datasets. In this work, we introduce ESS,
which tackles this problem by directly transferring the semantic segmentation
task from existing labeled image datasets to unlabeled events via unsupervised
domain adaptation (UDA). Compared to existing UDA methods, our approach aligns
recurrent, motion-invariant event embeddings with image embeddings. For this
reason, our method neither requires video data nor per-pixel alignment between
images and events and, crucially, does not need to hallucinate motion from
still images. Additionally, to spur further research in event-based semantic
segmentation, we introduce DSEC-Semantic, the first large-scale event-based
dataset with fine-grained labels. We show that using image labels alone, ESS
outperforms existing UDA approaches, and when combined with event labels, it
even outperforms state-of-the-art supervised approaches on both DDD17 and
DSEC-Semantic. Finally, ESS is general-purpose, which unlocks the vast amount
of existing labeled image datasets and paves the way for new and exciting
research directions in new fields previously inaccessible for event cameras.
