Reinventing 2D Convolutions for 3D Images
  There have been considerable debates over 2D and 3D representation learning
on 3D medical images. 2D approaches could benefit from large-scale 2D
pretraining, whereas they are generally weak in capturing large 3D contexts. 3D
approaches are natively strong in 3D contexts, however few publicly available
3D medical dataset is large and diverse enough for universal 3D pretraining.
Even for hybrid (2D + 3D) approaches, the intrinsic disadvantages within the 2D
/ 3D parts still exist. In this study, we bridge the gap between 2D and 3D
convolutions by reinventing the 2D convolutions. We propose ACS
(axial-coronal-sagittal) convolutions to perform natively 3D representation
learning, while utilizing the pretrained weights on 2D datasets. In ACS
convolutions, 2D convolution kernels are split by channel into three parts, and
convoluted separately on the three views (axial, coronal and sagittal) of 3D
representations. Theoretically, ANY 2D CNN (ResNet, DenseNet, or DeepLab) is
able to be converted into a 3D ACS CNN, with pretrained weight of a same
parameter size. Extensive experiments on several medical benchmarks (including
classification, segmentation and detection tasks) validate the consistent
superiority of the pretrained ACS CNNs, over the 2D / 3D CNN counterparts with
/ without pretraining. Even without pretraining, the ACS convolution can be
used as a plug-and-play replacement of standard 3D convolution, with smaller
model size and less computation.
