Joint 3D Object Detection and Tracking Using Spatio-Temporal
  Representation of Camera Image and LiDAR Point Clouds
  In this paper, we propose a new joint object detection and tracking (JoDT)
framework for 3D object detection and tracking based on camera and LiDAR
sensors. The proposed method, referred to as 3D DetecTrack, enables the
detector and tracker to cooperate to generate a spatio-temporal representation
of the camera and LiDAR data, with which 3D object detection and tracking are
then performed. The detector constructs the spatio-temporal features via the
weighted temporal aggregation of the spatial features obtained by the camera
and LiDAR fusion. Then, the detector reconfigures the initial detection results
using information from the tracklets maintained up to the previous time step.
Based on the spatio-temporal features generated by the detector, the tracker
associates the detected objects with previously tracked objects using a graph
neural network (GNN). We devise a fully-connected GNN facilitated by a
combination of rule-based edge pruning and attention-based edge gating, which
exploits both spatial and temporal object contexts to improve tracking
performance. The experiments conducted on both KITTI and nuScenes benchmarks
demonstrate that the proposed 3D DetecTrack achieves significant improvements
in both detection and tracking performances over baseline methods and achieves
state-of-the-art performance among existing methods through collaboration
between the detector and tracker.
