Class-Aware Generative Adversarial Transformers for Medical Image
  Segmentation
  Transformers have made remarkable progress towards modeling long-range
dependencies within the medical image analysis domain. However, current
transformer-based models suffer from several disadvantages: (1) existing
methods fail to capture the important features of the images due to the naive
tokenization scheme; (2) the models suffer from information loss because they
only consider single-scale feature representations; and (3) the segmentation
label maps generated by the models are not accurate enough without considering
rich semantic contexts and anatomical textures. In this work, we present
CASTformer, a novel type of generative adversarial transformers, for 2D medical
image segmentation. First, we take advantage of the pyramid structure to
construct multi-scale representations and handle multi-scale variations. We
then design a novel class-aware transformer module to better learn the
discriminative regions of objects with semantic structures. Lastly, we utilize
an adversarial training strategy that boosts segmentation accuracy and
correspondingly allows a transformer-based discriminator to capture high-level
semantically correlated contents and low-level anatomical features. Our
experiments demonstrate that CASTformer dramatically outperforms previous
state-of-the-art transformer-based approaches on three benchmarks, obtaining
2.54%-5.88% absolute improvements in Dice over previous models. Further
qualitative experiments provide a more detailed picture of the model's inner
workings, shed light on the challenges in improved transparency, and
demonstrate that transfer learning can greatly improve performance and reduce
the size of medical image datasets in training, making CASTformer a strong
starting point for downstream medical image analysis tasks.
