{'fraction_non_alphanumeric': 0.03734210373129984, 'fraction_numerical': 0.012078700739274697, 'mean_word_length': 5.089861751152074, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Class-Aware Generative Adversarial Transformers for Medical Image Segmentation

Transformers have made remarkable progress towards modeling long-range dependencies within the medical image analysis domain. However, current transformer-based models suffer from several disadvantages: (1) existing methods fail to capture the important features of the images due to the naive tokenization scheme; (2) the models suffer from information loss because they only consider single-scale feature representations; and (3) the segmentation label maps generated by the models are not accurate enough without considering rich semantic contexts and anatomical tex-tures. In this work, we present CASTformer , a novel type of generative adversarial transformers, for 2D medical image segmentation. First, we take advantage of the pyramid structure to construct multi-scale representations and handle multi-scale variations. We then design a novel class-aware transformer module to better learn the discriminative regions of objects with semantic structures. Lastly, we utilize an adversarial training strategy that boosts segmentation accuracy and correspondingly allows a transformer-based discriminator to capture high-level semantically correlated contents and low-level anatomical features. Our experiments demonstrate that CASTformer dramatically outperforms previous state-of-the-art transformer-based approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in Dice over previous models. Further qualitative experiments provide a more detailed picture of the model’s inner workings, shed light on the challenges in improved transparency, and demonstrate that transfer learning can greatly improve performance and reduce the size of medical image datasets in training, making CASTformer a strong starting point for downstream medical image analysis tasks. Codes and models will be available to the public.


Introduction
Accurate and consistent measurements of anatomical features and functional information on medical images can greatly assist radiologists in making accurate and reliable diagnoses, treatment planning, and post-treatment evaluation [1]. Convolutional neural networks (CNNs) have been the de-facto standard for medical image analysis tasks. However, such methods fail in explicitly modeling long-range dependencies due to the intrinsic locality and weight sharing of the receptive fields in convolution operations. Such a deficiency in context modeling at multiple scales often yields sub-optimal segmentation capability in capturing rich anatomical features of variable shapes and scales (e.g., tumor regions with different structures and sizes) [2,3]. Moreover, using transformers has been shown to be more promising in computer vision [4][5][6] for utilizing long-range dependencies than other, traditional CNN-based methods. In parallel, transformers with powerful global relation modeling abilities have become the standard starting point for training on a wide range of downstream medical imaging analysis tasks, such as image segmentation [6][7][8][9][10], image synthesis [11][12][13], and image enhancement [14][15][16][17].
Medical image semantic segmentation can be formulated as a typical dense prediction problem, which aims at performing pixel-level classification on the feature maps. Recently, Chen et al. [6] introduced TransUNet, which inherits the advantage of both UNet [18] and Transformers [4], to exploit high-resolution informative representations in the spatial dimension by CNNs and the powerful global relation modeling by Transformers. Although existing transformer-based approaches have proved promising in the medical image segmentation task, there remain several formidable challenges, because (1) the model outputs a single-scale and low-resolution feature representation; (2) prior work mainly adopts the standard tokenization scheme, hard splitting an image into a sequence of image patches of size 16×16, which may fail to capture inherent object structures and the fine-grained spatial details for the downstream dense prediction task; (3) compared to the standard convolution, the transformer architecture requires a grid structure, and thus lacks the capability to localize regions that contain objects of interest instead of the uninteresting background; and (4) existing methods are usually deficient in ensuring the performance without capturing both global and local contextual relations among pixels. We argue that transformer-based segmentation models are not yet robust enough to replace CNN-based methods, and investigate several above-mentioned key challenges transformer-based segmentation models still face.
Inspired by recent success of vision transformer networks [3,4,[19][20][21][22][23][24][25], we make a step towards a more practical scenario in which we only assume access to pre-trained models on public computer vision datasets, and a relatively small medical dataset, which we can use the weights of the pre-trained models to achieve higher accuracy in the medical image analysis tasks. These settings are particularly appealing as (1) such models can be easily adopted on typical medical datasets; (2) such a setting only requires limited training data and annotations; and (3) transfer learning typically leads to better performance [26][27][28][29]. Inspired by such findings, we propose several novel strategies for expanding its learning abilities to our setting, considering both multi-scale anatomical feature representations of interesting objects and transfer learning in the medical imaging domain.
First, we aim to model multi-scale variations by learning feature maps of different resolutions. Thus, we propose to incorporate the pyramid structure into the transformer framework for medical image segmentation, which enables the model to capture rich global spatial information and local multi-scale context information. Additionally, we consider medical semantic segmentation as a sequence-tosequence prediction task. The standard patch tokenization scheme in [4] is an art-splitting it into several fixed-size patches and linearly embedding them into input tokens. Even if significant progress is achieved, model performance is likely to be sub-optimal. We address this issue by introducing a novel class-aware transformer module, drawing inspiration from a progressive sampling strategy in image classification [30], to adaptively and selectively learn interesting parts of objects. This essentially allows us to obtain effective anatomical features from spatial attended regions within the medical images, so as to guide the segmentation of objects or entities.  Figure 2: An illustration of (a) our class-aware transformer module and (b) transformer network. Our class-aware transformer module iteratively samples discriminative locations. A group of points is initialized in the regularly-spaced location. At each step, given the feature map F i , we iteratively update its sampling locations by adding them with the estimated offsets of the last step. Note that only 4 points are shown for a clear presentation, and there are more points in the actual implementation.
Second, we adopt the idea of Generative Adversarial Networks (GANs) to improve segmentation performance and correspondingly enable a transformer-based discriminator to learn low-level anatomical features and high-level semantics. The standard GANs are not guaranteed to prioritize the most informative demonstrations on interesting anatomical regions, and mixing irrelevant regions (i.e., background) creates inferior contexts, which drastically underperform segmentation performance [2,3]. Additionally, it is well-known that they are notoriously difficult to train and prone to model collapse [31]. Training vision transformers is also tedious and requires large amounts of labeled data, which largely limits the training quality. We use a more refined strategy, where, for each input, we combine it with the predicted segmentation mask to create the image with anatomical demonstrations. We also leverage the pre-trained checkpoints to compensate the need of large-dataset training, thereby providing a good starting point with more discriminative visual demonstrations. We refer to our approach as CASTformer, class-aware generative adversarial transformers: a strong transformer-based method for 2D medical image segmentation. Our contributions are summarized as follows: • Novel Network Architecture: We make the first attempt to build a GAN using a transformer-based architecture for the 2D medical image segmentation task. We incorporate the pyramid structure into the generator to learn rich global and local multi-scale spatial representations, and also devise a novel class-aware transformer module by progressively learning the interesting regions correlated with semantic structures of images. To the best of our knowledge, we are the first work to explore these techniques in the context of medical imaging segmentation. • Better Understanding Inner Workings: We conduct careful analyses to understand the model's inner workings, how the sampling strategy works, and how different training factors lead to the final performance. We highlight that it is more effective to progressively learn distinct contextual representations with the class-aware transformer module, resulting in more accurate and robust models that applied better to a variety of downstream medical image analysis tasks (See Appendix I and J). • Remarkable Performance Improvements: CASTformer contributes towards a dramatic improvement across three datasets we evaluate on. For instance, we achieve Dice scores of 82.55% and 73.82% by obtaining gains up to 5.88% absolute improvement compared to prior methods on the Synapse multi-organ dataset. We illustrate the benefits of leveraging pre-trained models from the computer vision domain, and provide suggestions for future research that could be less susceptible to the confounding effects of training data from the natural image domain.
Transformers in Medical Image Segmentation Recent studies [6,7,9,10,[50][51][52][53][54][55] have focused on developing transformer-based methods for medical image analysis tasks. Recently, Chen et al. [6] proposed TransUNet, which takes advantage of both UNet and Transformers, to exploit highresolution informative information in the spatial dimension by CNNs and the global dependencies by Transformers. Cao et al. [7] explored how to use a pure transformer for medical image analysis tasks. However, the results do not lead to better performance. These works mainly utilized hard splitting some highly semantically correlated regions without capturing the inherent object structures. In this work, beyond simply using the naive tokenization scheme in [4,6], we aim at enabling the transformer to capture global information flow to estimate offsets towards regions of interest.
Transformer in Generative Adversarial Networks Adversarial learning has proved to be a very useful and widely applicable technique for learning generative models of arbitrarily complex data distributions in the medical domain. As the discriminator D differentiates between real and fake samples, the adversarial loss serves as the regularization constraint to enforce the generator G to predict more realistic samples. Inspired by such recent success [56][57][58][59][60][61], Jiang et al. [56] proposed to build a GAN pipeline with two pure transformer-based architectures in synthesizing high-resolution images. Esser et al. [22] first used a convolutional GAN model to learn a codebook of context-rich visual features, followed by transformer architecture to learn the compositional parts. Hudson et al. [3] proposed a bipartite self-attention on StyleGAN to propagate latent variables to the evolving visual features. Despite such success, it requires high computation costs due to the quadratic complexity, which fundamentally hinders its applicability to the real world. Besides the image generation task, we seek to take a step forward in tackling the challenging task of 2D medical image segmentation.

Method
Our proposed approach is presented in Figure 1. Given the input image x ∈ R H×W ×3 , similar to TransUNet architecture [6], our proposed generator network G, termed CATformer, is comprised of four key components: encoder (feature extractor) module, class-aware transformer module, transformer encoder module, and decoder module. As shown in Figure 1, our generator has four stages with four parallel subnetworks. All stages share a similar architecture, which contains a patch embedding layer, class-aware layer, and L i Transformer encoder layers.
Encoder Module. Our method adopts a CNN-Transformer hybrid model design instead of using a pure transformer, which uses 40 convolutional layers, to generate multi-scale feature maps. Such a convolutional stem setting provides two advantages: (1) using convolutional stem helps transformers perform better in the downstream vision task [62,63]; (2) it provides high-resolution feature maps with parallel medium-and low-resolution feature maps to help boost better representations. In this way, we can construct the feature pyramid for the Transformers, and utilize the multi-scale feature maps for the downstream medical segmentation task. With the aid of feature maps of different resolutions, our model is capable of modeling multi-resolution spatially local contexts.
Hierarchical Feature Representation. We deviate from TransUNet by generating a singleresolution feature map, and our focus is on extracting CNN-like multi-level features F i , where i ∈ {1, 2, 3, 4}, to achieve high segmentation accuracy by leveraging high-resolution features and low-resolution features. More precisely, in the first stage, we utilize the encoder module to obtain the dense feature map is the spatial feature resolution and the number of feature channels. In a similar way, we can formulate the following feature maps as follows: . Then, we divide F 1 into HW 16 2 patches with the patch size P of 16×16×3, and feed the flattened patches into a learnable linear transformation to obtain the patch embeddings of size HW 16 2 ×C 1 . Class-Aware Transformer Module. The class-aware transformer module (CAT) is designed to adaptively focus on useful regions of objects (e.g., the underlying anatomical features and structural information). Our CAT module is largely inspired by the recent success for image classification [30], but we deviate from theirs as follows: (1) we remove the vision transformer module in [30] to alleviate the computation and memory usage; (2) we use 4 separate Transformer Encoder Modules (TEM), which will be introduced below; (3) we incorporate M CAT modules on multi-scale representations to allow for contextual information of anatomical features to propagate into the representations. Our class-aware transformer module is an iterative optimization process. In particular, we apply the class-aware transformer module to obtain the sequence of tokens I M,1 ∈ R C×(n×n) , where (n × n) and M are the number of samples on each feature map and the total iterative number, respectively. As shown in Figure 2, given the feature map F 1 , we iteratively update its sampling locations by adding them with the estimated offset vectors of the last step, which can be formulated as follows: where s t ∈ R 2×(n×n) , and o t ∈ R 2×(n×n) are the sampling location and the predicted offset vector at t-th step. Specifically, the s 1 is initialized at the regularly spaced sampling grid. The i-th sampling location s i 1 is defined as follows: where The step sizes in the y (row index) and x (column index) directions denote τ h = H/n and τ w = W/n, respectively. · is the floor operation. We can define the initial token on the input feature map in the following form: , . . . , M }, and I t ∈ R C×(n×n) denotes the initial sampled tokens at t-th step. We set the sampling function as the bilinear interpolation, since it is differentiable with respect to both the sampling locations s t and the input feature map F i . We do an element-wise addition of the current positional embedding of the sampling locations, the initial sampled tokens, and the estimated tokens of the last step, and then we can obtain the output tokens at each step: where W t ∈ R C×2 is the learnable matrix that embeds s t to positional embedding S t ∈ R C×(n×n) , and ⊕ is the element-wise addition. Transformer(·) is the transformer encoder layer, as we will show in the following paragraphs. We can compute the estimated sampling location offsets as: where θ t (·) ∈ R 2×(n×n) is the learnable linear mapping for the estimated sampling offset vectors. It is worth noting that these operations are all differentiable, thus the model can be learned in an end-to-end fashion.
Transformer Encoder Module. The transformer encoder module (TEM) is designed to model long-range contextual information by aggregating global contextual information from the complete sequences of input image patches embedding. In implementations, the transformer encoder module follows the architecture in ViT [4], which is composed of Multi-head Self-Attention (MSA), and MLP blocks, which can be formulated as: where i = 1 . . . M , and LN(·) is the layer normalization. H ∈ R (P 2 ·C)×D and H pos ∈ R N ×D denote the patch embedding projection and the position embedding.
Decoder Module. The decoder is designed to generate the segmentation mask based on four output feature maps of different resolutions. In implementations, rather than designing a hand-crafted decoder module that requires high computational demand, we incorporate a lightweight All-MLP decoder [64], and such a simple design allows us to yield a powerful representation much more efficiently. The decoder includes the following criteria: 1) the channel dimension of multi-scale features is unified through the MLP layers; 2) we up-sample the features to 1/4th and concatenate  Discriminator Network. We use the R50+ViT-B/16 hybrid model pre-trained on ImageNet from ViT [4] as a starting point for our discriminator design, in this case using the pre-trained strategies to learn effectively on the limited size target task data. Then, we simply apply a two-layer multi-layered perceptron (MLP) to make a prediction about the identity of the class-aware image. Following previous work [2], we first utilize the ground truth image x and the predicted segmentation mask y to obtain the class-aware imagex (i.e., pixel-wise multiplication of x and y ). It is important to note that this construction re-uses the pre-trained weights and does not introduce any additional parameters. D seeks to classify between real and fake samples [65]. G and D compete with each other through attempting to reach an equilibrium point of the minimax game. Using this structure enables the discriminator to model long-range dependencies, making it better assess the medical image fidelity. This also essentially endows the model with a more holistic understanding of the anatomical visual modality (categorical features).
Training Objective. As to the loss function and training configurations, we adopt the settings used in Wasserstein GAN (WGAN) [66], and use WGAN-GP loss [67]. We jointly use the segmentation loss [6,10] and WGAN-GP loss to train G. Concretely, the segmentation loss includes the dice loss and cross-entropy loss. Hence, the training process of CASTformer can be formulated as: where λ 1 , λ 2 , λ 3 determine the importance of each loss term. See Appendix E for an ablation study.

Experimental Setup
Datasets. We experiment on multiple challenging benchmark datasets: Synapse 1 , LiTS, and MP-MRI. More details can be found in Appendix A.
Implementation Details. We utilize the AdamW optimizer [68] in all our experiments. For training our generator and discriminator, we use a learning rate of 5e −4 with a batch size of 6, and train each model for 300 epochs for all datasets. We set the sampling number n on each feature map and the total iterative number M as 16 and 4, respectively. See Appendix E, F, G for details on hyperparameters, the training configuration and model architecture. We also adopt the input resolution and patch size P as 224×224 and 14, respectively. We set λ 1 = 0.5, λ 2 = 0.5, and λ 3 = 0.1 in this experiments.
In the testing stage, we adopt four metrics to evaluate the segmentation performance: Dice coefficient Experiments: Synapse Multi-organ. The quantitative results on the Synapse dataset are shown in Table 1. The results are visualized in Figure 3. It can be observed that our CATformer outperforms the previous best model by a large margin and achieves a 4.69% − 8.44% absolute improvement in Dice and Jaccard, respectively. Our CASTformer achieves the best performance of 82.55% and 74.69%, dramatically improving the previous state-of-the-art model (TransUNet) by +5.07% and +9.91%, in terms of both Dice and Jaccard scores. This shows that the anatomical visual information is useful for the model to gain finer control in localizing local semantic regions. As also shown in Table 1, our CASTformer achieves absolute Dice improvements of +2.77%, +2.51%, +1.35%, +4.95% on large organs (i.e., left kidney, right kidney, liver, stomach) respectively. Such improvements demonstrate the effectiveness of learning the evolving anatomical features of the image, as well as accurately identifying the boundary information of large organs. We also observed similar trends that, compared to the previous state-of-the-art results, our CASTformer obtains 89.05%, 67.48%, 67.49% in terms of Dice on small organs (i.e., aorta, gallbladder, pancreas) respectively, which yields big improvements of +1.82%, +0.95%, +10.91%. This clearly demonstrates the superiority of our models, allowing for a spatially finer control over the segmentation process.  Figure 4, our method is capable of predicting high-quality object segmentation, considering the fact that the improvement in such a setting is challenging. This demonstrates: (1) the necessity of adaptively focusing on the region of interests; and (2) the efficacy of semantically correlated information. Compared to previously high-performing models, our two approaches achieve significant improvements on all datasets, demonstrating their effectiveness.

Analysis
In this section, we conduct thorough analyses of our CASTformer in the following aspects: transfer learning, model components, effects of iteration number N (Appendix C), sampling number n (Appendix D), hyperparameter selection (Appendix E), different GAN-based loss functions (Appendix H), and gain a better understanding of the model's inner workings (Appendix I and J).
Transfer Learning. We consider whether we can leverage the pre-trained model commonly used in computer vision literature [4], to provide more evidence for the beneficial impact of our network performance. We use CATformer (G) as the baseline and evaluate all the settings on the Synapse multi-organ dataset. To put our results in perspective, we compare with six ways of using pre-trained R50+ViT-B/16 hybrid model from ViT [4] for transfer learning, namely (1) CATformer: w/o pretrained and w/ pre-trained; (2) CASTformer: both w/o pre-trained, only w/ pre-trained D, only w/ pre-trained G, and both w/ pre-trained G and D.  The results are in Table 3. Overall, we observe that using "w/ pre-trained" leads to higher accuracy than "w/o pre-trained", with significant improvements for the smaller sizes of datasets, suggesting that using "w/ pre-trained" provides us a good set of initial parameters for the downstream tasks. With using pre-trained weights, CATformer outperforms the setting without using pre-trained weights by a large margin and achieves 7.33% and 7.61% absolute improvements in terms of Dice and Jaccard. CASTformer ("both w/ pre-trained") also yields big improvements (+8.91% and +12.01%) in Dice and Jaccard. This suggests that CASTformer is better at both initializing from the pre-trained models and better at gathering the anatomical information in a more adaptive and selective manner. As shown in Table 3, surprisingly, there is a significant discrepancy between only using "w/ pre-trained D" and "w/ pre-trained G": for example, CASTformer achieves 78.87% in Dice with only w/ pre-trained D, while CASTformer achieves 81.46% if only G uses the pre-trained weights. This demonstrates that only using pre-trained weight in D might be the culprit for the exploitation of anatomical information.
Our results suggest that (1) utilizing pre-trained models in the computer vision domain can help the model quickly adapt to new downstream medical segmentation tasks without re-building billions of anatomical representations; (2) we find that leveraging pre-trained weights can further boost the performance because it can mitigate the discrepancy between training and inference; and (3) it also creates a possibility to adapt our model to the typical medical dataset with the smaller size.

Ablation of Model Components.
Our key observation is that it is crucial to build high-quality anatomical representations through each model component. To show the strengths of our approach, we examine the following variants and inspect each key component on the Synapse multi-organ segmentation dataset: (1) Baseline: we remove the class-aware transformer module and the transformer encoder module in our CATformer as the baseline, similar to TransUNet defined in [6]; (2) CATformer w/o CAT: we only remove the class-aware transformer module in our CATformer; (3) CATformer w/o TEM: we only remove the transformer encoder module in our CATformer; (4) CATformer: this is our G model; and (5) CASTformer: this is our final model described in Section 3. Table 4 summarizes the results of all the variants. As is shown, we observe that compared to the baseline model, both CATformer w/o CAT and CATformer w/o TEM are able to develop a better holistic understanding of global shapes/structures and fine anatomical details, thus leading to large performance improvements (+2.61% and +3.87%) in terms of Dice. Our results show the class-aware transformer module is useful in improving the segmentation performance, suggesting that the discriminative regions of medical images are particularly effective. Finally, one thing worth noticing is that incorporating both the class-aware transformer module and the transformer encoder module performs better than only using a single module, highlighting the importance of two modules in our CATformer.

Conclusion and Discussion of Broader Impact
In this work, we have introduced CASTformer, a simple yet effective type of generative adversarial transformers, for 2D medical image segmentation. The key insight is to integrate the multi-scale pyramid structure to capture rich global spatial information and local multi-scale context information. Furthermore, CASTformer also benefits from our proposed class-aware transformer module to progressively and selectively learn interesting parts of the objects. Lastly, the generator-discriminator design is used to boost segmentation performance and correspondingly enable the transformer-based discriminator to capture low-level anatomical features and high-level semantics. Comprehensive experiments demonstrate that our CASTformer outperforms the previous state-of-the-art on three popular medical datasets considerably. We conduct extensive analyses to study the robustness of our approach, and form a more detailed understanding of desirable properties in the medical domain (i.e., transparency and data efficiency).
Overall, we hope that this model can serve as a solid baseline for 2D medical image segmentation and motivate further research in medical image analysis tasks. It also provides a new perspective on transfer learning in medical domain, and initially shed novel insights towards understanding neural network behavior. As such pattern is hard to quantify, we expect more mechanistic explanations for clinical practise. We also plan to optimize the transformer-based architectures for the downstream medical image analysis tasks both in terms of data and model parameters.
Broader Impact. We acknowledge that our research will not pose significant risks of societal harm to society. Our work is scientific nature and will have the potential to positively contribute to a number of real-world clinical applications that establish high-quality and end-to-end medical image segmentation systems. We expect our approach to contribute to the grand goal of building more secured and trustworthy clinical AI.

B More Experiments: MP-MRI
Experimental results are summarized in Table 5. Overall, CATformer and CASTformer outperform the previous results in terms of Dice and Jaccard. Compared to SETR, our CATformer and CASTformer perform 1.78% and 2.54% higher in Dice, respectively. We also find CASTformer performs better than CATformer, which suggests that using discriminator can make the model better assess the medical image fidelity. Figure 5 shows qualitative results, where our CATformer and CASTformer provide better anatomical details than all other methods. This clearly demonstrates the superiority of our models. All these experiments are conducted using the same hyperparameters in our CASTformer.

C Effect of Iteration Number N
We explore the effect of different iteration number N in Figure 6 (a). Note that in the case of N = 1, the sampling locations will not be updated. We find that more iterations of sampling clearly improve network performance in Dice and Jaccard. However, we observe that the network performance does not further increase from N = 4 to N = 6. In our study, we use N = 4 for the class-aware transformer module. We further evaluate the effect of sampling number n of the class-aware transformer module in Figure  6 (b). Empirically, we observe that results are generally well correlated when we gradually increase the size of n. As is shown, the network performance is optimal when n = 16.
(a) (b) Figure 6: Effects of the iteration number N and the sampling number n in the class-aware transformer module. We report Dice and Jarrcd of CATformer on the Synapse multi-organ dataset.

F More Implementation Details
The training configuration and hyperparameter settings are summarized in Table 6.

G Model Architecture
We present the detailed architecture of CATformer's encoding pipeline in Section 7. We use input/output names to indicate the direction of the data stream. CATformer applies independent class-aware attention on 4 levels of features extracted by the ResNetV2 model. Each feature level L-k is processed by CATformer-k, consisting of 4 blocks of class-aware transformer modules, followed by 12 layers of transformer encoder modules. Outputs from all four feature levels are fed into the decoder pipeline to generate the segmentation masks.

H Importance of Loss Functions
One main argument for the discriminator is that modeling long-range dependencies and acquiring a more holistic understanding of the anatomical visual information can contribute to the improved capability of the generator. Besides the WGAN-GP loss [67], the minimax (MM) GAN loss [65], the Non-Saturating (NS) GAN loss [76], and Least Squares (LS) GAN Loss [77] are also commonly used as adversarial training. We test these alternatives and find that, in most cases, using WGAN-GP loss achieves comparable or higher performance than other loss functions. In addition, models trained using MM-GAN loss perform comparably to those trained using LS-GAN loss. In particular, our approach outperforms the second-best LS-GAN loss [77] by 1.10 and 2.49 points in Dice and Jaccard scores on the Synapse multi-organ dataset. It demonstrates the effectiveness of the WGAN-GP loss in our CASTformer.  [65]; NS-GAN loss [76]; LS-GAN loss [77]; and WGAN-GP loss [67].

I Visualization of Learned Sampling Location
To gain more insight into the evolving sampling locations learned by our proposed class-aware transformer module, we visualize the predicted offsets in Figure 8. We can see that particular sampling points around objects tend to attend to coherent segmented regions in terms of anatomical similarity and proximity. As is shown, we show the classes with the highly semantically correlated regions, indicating that the model coherently attends to anatomical concepts such as liver, right/left kidney, and spleen. These visualizations also illustrate how it behaves adaptively and distinctively to focus on the content with highly semantically correlated discriminative regions (i.e., different organs). These findings can thereby suggest that our design can aid the CATformer to exercise finer control emphasizing anatomical features with the intrinsic structure at the object granularity. As is indicated (Figure 8 last column), we also find evidence that our model is prone to capture some small object cases (e.g., pancreas, aorta, gallbladder). We hypothesize that it is because they contain more anatomical variances, which makes the model more difficult to exploit.

J Vision Transformer Visualization
In this section, we visualize the first 12 class-aware transformer layers on sequences of 28×28 feature patches in the encoder pipeline. In Figure 9, we plot the attention probabilities from a single patch over different layers and heads. Each row corresponds to one CAT layer; each column corresponds to an attention head. As we go deeper into the network, we are able to observe three kinds of attention behaviors as further discussed below. Attend to similar features: In the first group of layers (layer 1 through 4), the attention probability is spread across a relatively large group of patches. Notably, these patches correspond to areas in the image with similar color and texture to the query patch. These more primitive attention distributions indicate that the class-awareness property has not yet been established.
Attend to the same class and its boundary: In the middle layers of the transformer model, most noticeable in the 5 th and 6 th layers, the attention probabilities start to concentrate on areas that share the same class label as the query patch (layer 5-2). In some other instances, the model attends to the boundary of the current class (layer 5-3, 5-6).