Linear Context Transform Block
  Squeeze-and-Excitation (SE) block presents a channel attention mechanism for
modeling global context via explicitly capturing dependencies across channels.
However, we are still far from understanding how the SE block works. In this
work, we first revisit the SE block, and then present a detailed empirical
study of the relationship between global context and attention distribution,
based on which we propose a simple yet effective module, called Linear Context
Transform (LCT) block. We divide all channels into different groups and
normalize the globally aggregated context features within each channel group,
reducing the disturbance from irrelevant channels. Through linear transform of
the normalized context features, we model global context for each channel
independently. The LCT block is extremely lightweight and easy to be plugged
into different backbone models while with negligible parameters and
computational burden increase. Extensive experiments show that the LCT block
outperforms the SE block in image classification task on the ImageNet and
object detection/segmentation on the COCO dataset with different backbone
models. Moreover, LCT yields consistent performance gains over existing
state-of-the-art detection architectures, e.g., 1.5$\sim$1.7% AP$^{bbox}$ and
1.0$\sim$1.2% AP$^{mask}$ improvements on the COCO benchmark, irrespective of
different baseline models of varied capacities. We hope our simple yet
effective approach will shed some light on future research of attention-based
models.
