Modeling Point Clouds with Self-Attention and Gumbel Subset Sampling
  Geometric deep learning is increasingly important thanks to the popularity of
3D sensors. Inspired by the recent advances in NLP domain, the self-attention
transformer is introduced to consume the point clouds. We develop Point
Attention Transformers (PATs), using a parameter-efficient Group Shuffle
Attention (GSA) to replace the costly Multi-Head Attention. We demonstrate its
ability to process size-varying inputs, and prove its permutation equivariance.
Besides, prior work uses heuristics dependence on the input data (e.g.,
Furthest Point Sampling) to hierarchically select subsets of input points.
Thereby, we for the first time propose an end-to-end learnable and
task-agnostic sampling operation, named Gumbel Subset Sampling (GSS), to select
a representative subset of input points. Equipped with Gumbel-Softmax, it
produces a "soft" continuous subset in training phase, and a "hard" discrete
subset in test phase. By selecting representative subsets in a hierarchical
fashion, the networks learn a stronger representation of the input sets with
lower computation cost. Experiments on classification and segmentation
benchmarks show the effectiveness and efficiency of our methods. Furthermore,
we propose a novel application, to process event camera stream as point clouds,
and achieve a state-of-the-art performance on DVS128 Gesture Dataset.
