MOPT: Multi-Object Panoptic Tracking
  Comprehensive understanding of dynamic scenes is a critical prerequisite for
intelligent robots to autonomously operate in their environment. Research in
this domain, which encompasses diverse perception problems, has primarily been
focused on addressing specific tasks individually rather than modeling the
ability to understand dynamic scenes holistically. In this paper, we introduce
a novel perception task denoted as multi-object panoptic tracking (MOPT), which
unifies the conventionally disjoint tasks of semantic segmentation, instance
segmentation, and multi-object tracking. MOPT allows for exploiting pixel-level
semantic information of 'thing' and 'stuff' classes, temporal coherence, and
pixel-level associations over time, for the mutual benefit of each of the
individual sub-problems. To facilitate quantitative evaluations of MOPT in a
unified manner, we propose the soft panoptic tracking quality (sPTQ) metric. As
a first step towards addressing this task, we propose the novel
PanopticTrackNet architecture that builds upon the state-of-the-art top-down
panoptic segmentation network EfficientPS by adding a new tracking head to
simultaneously learn all sub-tasks in an end-to-end manner. Additionally, we
present several strong baselines that combine predictions from state-of-the-art
panoptic segmentation and multi-object tracking models for comparison. We
present extensive quantitative and qualitative evaluations of both vision-based
and LiDAR-based MOPT that demonstrate encouraging results.
