{'fraction_non_alphanumeric': 0.03487376979032948, 'fraction_numerical': 0.01084010840108401, 'mean_word_length': 4.9938020944646295, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Pyramid Medical Transformer for Medical Image Segmentation

Deep neural networks have been a prevailing technique in the field of medical image processing. However, the most popular convolutional neural networks (CNNs) based methods for medical image segmentation are imperfect because they model long-range dependencies by stacking layers or enlarging filters. Transformers and the self-attention mechanism are recently proposed to effectively learn long-range dependencies by modeling all pairs of word-to-word attention regardless of their positions. The idea has also been extended to the computer vision field by creating and treating image patches as embeddings. Considering the computation complexity for whole image self-attention, current transformer-based models settle for a rigid partitioning scheme that potentially loses informative relations. Besides, current medical transformers model global context on full resolution images, leading to unnecessary computation costs. To address these issues, we developed a novel method to integrate multi-scale attention and CNN feature extraction using a pyramidal network architecture, namely Pyramid Medical Transformer (PMTrans). The PMTrans captured multi-range relations by working on multi-resolution images. An adaptive partitioning scheme was implemented to retain informative relations and to access different receptive fields efficiently. Experimental results on three medical image datasets (gland segmentation, MoNuSeg, and HECKTOR datasets) showed that PMTrans outperformed the latest CNN-based and transformer-based models for medical image segmentation.

multiple scales [7] because they arbitrarily divide images. For example, TransFuse [5] uses patchwise attention, and the patches predetermine fixed receptive fields on the original image. Similarly, Medical Transformer [1] divides an image into 16 pieces for processing. It computes pixel-wise axial attention [1] within a piece, and each piece also has a fixed receptive field on the image; doing so may even separate the same object into different pieces, making segmentation difficult.
Furthermore, arbitrarily dividing images does not consider objects of different shapes and sizes. Intuitively, the position and size of an object inherently determine if it occupies the whole patch (large object), is contained in one patch (small object), or is divided into several patches (an object on cutting boundaries). Neighboring pixels often provide vital information of local details [7], whereas such a dividing scheme may destroy these relations, adversely affecting segmentation outcomes.
In short, while they have been demonstrated to outperform convolutional neural networks (CNNs) based methods, the existing transformer-based methods for medical image segmentation are far from reaching their maximal potential. The primary culprit of their inefficacy is the rigid partitioning an image into patches with fixed sizes, which is doomed to fail to grasp associations of diverse and unknown scales within the image.
We propose a novel, pyramid medical transformer (PMTrans) approach that explores and incorporates multi-resolution attention. To our knowledge, this is the first multi-scale and multiresolution medial image transformer, which is not only designed for medical images but also applicable to general images such as scenery pictures.
The new method has three prominent features. Firstly, the new model uses a pyramid architecture with four branches: three transformer branches and one CNN branch. The three transformers work on three different image scales to capture different ranges of correlations within the image, i.e., short-range, mid-range, and long-range associations. We use down-sampled images (half and 1/4 sizes of the original) to capture mid-range and long-range relations and use the original image to capture local, finer details. This pyramid design not only improves the efficiency of global context modeling but also accommodates objects with diverse sizes and shapes robustly.
Secondly, we abandon the existing partitioning schemes [1,4,5,8] and adopt gated axial attention and CNN as building blocks. We introduce local gated axial attention layers [1] in the short-range transformer. We only compute gated axial attention within a certain span, which is much shorter than the height or width of the original image. This local gated axial attention module does not concern global attention because pixels only attend to their neighbors. We focus on local finer details in this short-range transformer, and the global context is ignored for efficiency. We use global gated axial attention layers on downsampled input images in the mid-range and long-range branches to capture global attention effectively. Since global axial attention may be relatively costly to compute, we apply it to down-sampled images, which means we cover larger receptive fields on the original image at low costs.
Thirdly, we design and implement a new fusion and label guidance scheme to integrate feature maps of different scales from the three multi-scale transformers and image-specific features from the CNN branch. We preserve low-level context without deep CNNs and retain the feature extraction characteristics of CNN and transformer models [5]. As for label guidance, we inject deep supervision [2] into different feature map scales in the training process, which stabilizes the training and alleviates a potential gradient vanishing problem [9].
We tested the performance of the proposed PMTrans model on three datasets: gland segmentation (microscopic) dataset [10], MoNuSeg (microscopic) dataset [11], and HECKTOR (PET/CT) dataset [12]. The results showed that the new model outperformed the latest CNN-based and transformer-based models.

Related Work
Before deep learning was introduced to the medical field, most automated medical image segmentation models are atlas-based [13,14], statistical-based [15], and shape-based [16]. With the great successes of CNNs, particularly on images, many CNN-based medical image segmentation models have been developed. The most popular is the encoder-decoder architecture proposed in the U-net [17], subsequently improved and extended. Some architectures replace the vanilla stacking convolutional layers with other backbones, resulting in, e.g., residual U-net [18] and dense U-net [19] as well as their implementation innovations, such as U-net++ [20], V-net [21], Y-net [22]. These CNN-based methods work on various modalities, target locations, and segmented targets. They not only perform well on a wide range of images but also expand the capacity of convolutional neural networks.
Although most latest computer vision models use CNNs as building blocks, the drawback of CNNs is also noticeable and profound. Convolutional layers are not designed to capture long-range dependencies because they aggregate local information within their filter regions across one layer to the next; capturing long-range associations requires deep networks or very large filters, resulting in models with a large number of parameters that are difficult and costly to train. A few approaches have been proposed to address this issue: 1) using the atrous convolution [7] so that the receptive fields enlarge without increasing the number of parameters; 2) incorporating statistical modeling like Markov Random Field to support global dependencies [6]; 3) using a feature pyramid structure to deal with multiple image resolutions [23].
Transformer [24] is a ground-breaking deep-learning technique for self-attention. It makes the selfattention mechanism practically feasible at a global scale so that long-range dependencies can be learned efficiently. Transformer-based models revolutionize the natural language processing (NLP) field, reminiscent of CNNs revolutionizing the field of computer vision (CV). Effective transformer models like BERT [25] and GPT [26] outperform the previous sequence-to-sequence NLP models in many tasks like machine translation and text generation.
The idea of the transformer has been attempted in developing attention-based CV models. One major difficulty of adopting transformers for CV tasks is the high computational complexity and cost for modeling and learning global attention. In NLP tasks, sentences are normally less than 500 words, which is manageable for computing all pairwise attention of the words in a sentence. However, for an by image, pixels give rise to a quadratic order of (i.e., 2 2 ) pairs of attention weights, e.g., more than 68 billion pairs of attention for a 512 by 512 image. This direct adaptation of the idea for images is not scalable. To address this problem, Vision Transformer (ViT) [24] partitions a 2D image into patches and then converts the patches with a positional encoding to a 1D sequence to feed into transformer blocks. Instead of computing pixel-to-pixel attention, it computes patch-to-patch attention. For an by image with a × patch size, this scheme reduces to / 2 patches and 2 2 / 4 pairs of attention weights. i.e., roughly one million pairs of attention for a 512 by 512 image with a patch size of 16 by 16. Segmentation Transformer [8] replaces the encoder in a CNN-based method with a transformer with superior performance for end-to-end segmentation tasks.
Beyond using patches to reduce the number of parameters of self-attention, approximation alternatives have been considered to avoid the quadratic computational cost. Learning global selfattention amounts to computing all pairwise relations. Nevertheless, full attention is not the only solution. Axial-Deeplab [27] introduces an approximation method that substitutes vanilla global attention with axial attention by computing the vertical and horizontal axial attention, which reduces the complexity from ( 2 2 ) to ( ) where m is the axial span of the attention operation. Sparse transformer [28] extends this approximation method and generates long sequences to reduce the complexity from ( 2 2 ) to ( √ ).
Besides high computation cost, the second difficulty that vision transformers encounter is that they require colossal datasets for pre-training. Randomly initiated model weights let each patch attend to any other patch randomly at the start of the training, so a large amount of training samples is needed to learn where to "look" and aggregate information to pass onto the next layer. Compared to CNN-based models, where filters slide through the image stride by stride, the self-attention mechanism offers more liberty to attend yet facing higher learning costs. Data-efficient Transformer (DeiT) [29] addresses this issue with a distillation mechanism, making the model trainable on mid-sized datasets.
In the medical image segmentation field, transformer-based methods have not yet been fully explored. High-performance transformers for medical images face two bottlenecks: the computational cost and lack of sufficient training data. In clinical practice, high computational cost posts a burden on the hardware requirements and increases the patient's wait time which may not be acceptable. Moreover, medical datasets are typically much smaller than daily images. ViT [24] uses more than 300M regular images to pre-train the model. Nevertheless, none of the currently available medical datasets has a sufficient amount of training images because medical data collection and annotation demand medical expertise [2].
The current medical image segmentation transformers fall into two categories. The first integrates the attention mechanism into their CNN models as a performance-boosting component, as done in Attention U-net [30] and volumetric attention model [31]. The second type of model is mainly built with transformer layers and uses CNNs as feature extraction blocks. For example, TransUnet [4] inserts attention layers at the end of the CNN-based feature extraction encoder, and the model relies on the pre-trained weights of ViT. Medical Transformer [1] uses gated axial attention and a local-global (LoGo) strategy, where the gated axial attention module is to address the lack-of-data issue, and the LoGo strategy reduces the computation cost of global self-attention. TransFuse [5] combines feature maps from a CNN and a transformer. However, the rigid patching scheme hinders the performance of all previous medical transformers, but without it they would suffer from drastically increased computation cost. Our approach effectively avoids the patching strategy, and wisely applies small axial attention window at multiple scales, capturing dependencies of all ranges efficiently.

Overview
Our approach to capturing both global and local relations, with a manageable computation cost, is to build a pyramid deep-learning architecture, which we call the pyramid medical transformer (PMTrans, Fig. 1). The new method has four branches: three transformer branches are built with gated axial attention blocks [1], and the fourth branch with CNN blocks. Fig. 1 Structure Overview. The pyramid medical transformer has three transformer branches (short, mid, and long-range branches) and a CNN branch. Input images are rescaled to ½ and ¼ for mid-range and longrange branches, respectively. Feature maps from the transformer branches (in purple) are fused with those feature maps from the CNN branches (in green) at different scales. Deep supervisions are injected at ½ and ¼ scales during the training phase.
The three transformer branches ( Fig. 1) are designed to capture short-range, mid-range, and longrange associations among the pixels of an image. They have similar structures but take three different scales of the image ( ℎ × ℎ × ℎ ) as input, which are × × , 2 × 2 × , and 4 × 4 × , respectively. They all have an initial convolution block for basic feature extraction at the beginning. This convolution block has three convolution layers followed by batch normalization and the ReLU function [1]. They then have an encoder-decoder-style transformer block to compute self-attention among the pixels. These transformer blocks are built with gated axial attention layers [1] and have skip connections between the encoder and decoder paths. The short-range transformer block has a depth of five, the mid-range block of four, and the long-range block of three accommodating the different sizes of their input feature maps. The outputs of these three branches have the same height and width as their inputs and are subsequently integrated with the feature maps from the CNN branch. The combined results are then up-sampled to restore the original resolution. Using multi-scale input images improves the segmentation of objects with different sizes and shapes.
We design the fourth branch with CNN-based residual blocks (Fig. 1) for extracting features in the input image. Comparing with CNN-based models, we only use a shallow network for the task. By fusing the feature maps from the transformers and CNN, our model preserves low-level context and global relations. Deep supervision [9] is introduced to the two down-sampled feature maps. We build two auxiliary classification layers at the two down-sampled resolutions and use the down-sampled ground-truth label maps correspondingly to stabilize the training process and alleviate the potential gradient vanishing problem [9].

Gated axial attention
In the three transformer branches, we use the gated axial attention, an extension to axial attention [1]. Axial attention computes approximate self-attention. Consider an input image or a feature map X. The self-attention Y of X is computed as where Q, K, and V are query, key, and value matrices computed by weight , , and , respectively; is the number of dimensions of the keys and queries [32]. Note that dividing by √ is to counteract the effect of having small gradients due to large dot products because of high-dimensional keys and queries [32]. Assume that we have a feature map ∈ × × , where , , and represent the height, width, and number of channels, respectively. There are × tokens and each of which has dimensions. We need to compute the attention between each super-pixel and all tokens on the feature map, i.e., a total of 2 2 pairs of attention to compute. Self-attention is effective for capturing relations of any distance. All pairs of attention have equal opportunities to be selected by the model regardless of how far two tokens are apart from each other. However, an obvious drawback of self-attention is the overall computational cost.
As an approximation, axial attention [33] applies self-attention computation along the height and width axes of the image. It first computes the attention between each pixel and all pixels in the same column (i.e., along the height) and then the attention between each pixel and all pixels in the same row (i.e., along the width). Note that the attention along the height axis aggregate information in the same column, and subsequently, the attention along the width axis aggregates the combined information in the same row. When each pixel attends to all pixels in the same row, it also indirectly or approximately attends to all other pixels in the feature map. Doing so reduces the computation from × to × ( + ). For an input feature map ∈ × × , the updated self-attention mechanism [1] with positional encodings along the width axis is: where and are the pixel positions along the width and height axes; and , , and are relative position encodings regarding queries, keys, and values, respectively. Note that the relative position embedding is also along the width axis. The axial attention along the height axis is similar. Gated axial attention [1] adds weights (i.e., gates) to the relative position encodings in Eqn. (3). These gates would reduce the effect of these encodings if the model cannot learn accurate position encodings with small medical datasets. The gated axial attention computes where , , 1 , and 2 are gates that control the influence of position encodings for the queries, keys, and values [1], respectively. Our model builds the encoders of the transformers with gated axial attention layers (Figure 2). Each of the encoder blocks has a 1 × 1 convolution layer, two multi-head axial attention layers (one each along the height and width axes), and one 1×1 convolution layer (Fig. 2). Residual connection is also included in encoder blocks. A decoder block consists of a series of one convolution layer, one bilinear up-sampling layer, and one ReLU layer. The input to the short-range branch is the original image ( × × ). It uses local gated axial attention blocks to model local context. We set the span of the gated axial attention to 4 and 4 .
Each pixel attends to a subregion ( 4 × 4 ) centered on itself (Fig. 3). We avoid the trivial partitioning scheme that would potentially cut off valuable relations. Compared to the patching solution proposed by MedTrans [1], which partitions the original image into 4 × 4 patches, the gated axial attention that we use has the same complexity, yet our partitioning scheme is adaptive.  [1], which involves ( + ) × pairs of attention, this long-range branch only has 1/216 of the attention pairs used in MedTrans. We stack three layers in this encoder-decoder architecture.
In summary, the three transformer branches of the PMTrans model have different receptive fields on the input image, but all use fixed-range gated axial attention. With this pyramid architecture, we model relations at three different resolutions and ranges efficiently. This design not only effectively caters to objects at different scales but also computes global attention without computing full resolution self-attention. Moreover, we avoid arbitrarily partitioning the image into patches. Instead, we preserve a pixel's local relations by computing axial self-attention within a patch centered on itself.

Fusion and deep supervision
We introduce the fourth, CNN branch to the model to extract features in the input image (Fig. 1). This branch is built with residual convolution blocks, stacking convolution layers with skip connections for gradient flow enhancement [34]. Feature maps from the CNN branch are fused into those from the transformer branches at different scales (Fig. 1), and the pyramid fusion scheme is built with attention gates proposed by the attention U-net [30]. Instead of simply concatenating feature maps and let convolution layers learn to fuse them, we add weights to integrate them consciously with attention-aware learnable parameters at different scales. Using these attention gates at multiple scales enables the model to extract relevant information from both transformers and CNNs, exploiting the feature extraction and information aggregation capabilities of both techniques [35]. We introduce deep supervision by adding auxiliary classifiers after the pyramid fusion of feature maps. Label guidances at different scales are provided at these three branches during training, and only the full resolution output is used for testing and evaluation. These deep supervisions enforce intermediate feature maps to be semantically discriminative at each scale [36].
• The GLAS dataset contains 85 microscopic images for training and 80 for testing, each with a resolution of 128 by 128 [10]. • The MoNuSeg dataset has 30 microscopic images for training and 14 images for testing, each with a resolution of 1000 by 1000 [11]. • The HECKTOR dataset contains 201 training cases and 53 testing cases, each of which has 144 × 144 × 144 CT/PET volumes [12]. We stacked and fused the CT and PET slices to feed into the network.
We set the training/validation split for 65/20, 25/5, and 180/21 for GLAS, MoNuSeg, and HECKTOR, respectively. We applied random shifting, rotation, and flipping for data augmentation. We performed cross-validation on both datasets to select the best model weights.

Training Details
The Pyramid Medical Transformer was implemented with Pytorch and trained on an RTX 3090 GPU. We set the batch size to 2 and the learning rate to 0.001 with a polynomial learning rate scheduler. The model was trained using Adam optimizer [37], and the training process finished within 400-500 epochs. The binary cross-entropy loss was defined as where ̂ is the prediction label map and is the ground truth label map. As suggested by [1], we did not activate the gates' training in the first 10 epochs. We used the binary cross-entropy loss for training and evaluated the final models by Dice Similarity Coefficient (DSC), defined as where | | and | | stand for the numbers of pixels in the ground-truth and predicted masks, and | ∩ | is the number of pixels in the overlapping region [6].

Results
We carried out a quantitative comparison of PMTrans with several state-of-the-art medical image segmentation methods, including both CNN-based methods (FCN [38], U-net [17], U-net++ [20], and Res-Unet [34]) and attention-based methods (Axial Attention U-net [27] and Medical Transformer [1]). The results are shown in Table 1. Note that all results on the HECKTOR dataset are based on the validation set because we do not have access to the ground truth of the test set.
Our PMTrans method outperformed all six baseline methods on both datasets (Table 1). PMTrans outperformed the best convolution-based baseline by 3.36% on the GLAS dataset, 0.75% on the MoNuSeg dataset, and 1.41% on the HECKTOR dataset. PMTrans surpassed the best transformerbased baseline method (MedTrans) by 0.57%, 0.68%, and 2.21% on the GLAS, MoNuSeg, and HECKTOR datasets, respectively. We also visually compared the segmentation outcomes of these methods using the test images. Figure 4  We studied the effects of the new network structure and an adaptive partitioning mechanism ( Figure 5). We evaluated the performance of combining only one of the short, mid, and long-range attention with the image features extracted by a CNN branch. For comparison, we tested the partitioning scheme used by MedTrans [1], which means we partition the feature maps into patches and perform "global" axial attention on H/4×W/4 patches. We kept the rest structure of the model and training parameters. The major difference between the partitioning scheme and our new method is that each pixel attends to a patch centered on itself instead of a pre-cut patch.
The results showed that on the GLAS and HECKTOR dataset, shorter-range branches have lower performance than long-range branches, and the PMTrans model that comprises three branches has the best performance. The difference between the short-range branch and the long-range branch is 7.52%. This effect can also be seen on the MoNuSeg dataset, but it is less obvious. The difference between the short-range branch and the long-range branch is 1.44%. The effectiveness of the nontrivial partitioning scheme varies due to the different sizes of the segmentation object from the three datasets. The gland segmentation targets are considerably larger than the nucleus, and it is more likely that a 4 × 4 patch cannot contain one object, which means the model needs to learn dependencies across longer ranges. However, for the MoNuSeg dataset, each nucleus is much smaller than a patch, so the difference between shorter-range land longer-range branches is not as significant. Moreover, we showed that the adaptive partitioning scheme improved the model performance on both datasets. CNN-branch is integrated into the model to exploit the feature extraction power of convolutional layers. We tested three different CNN structures as well as removing the CNN branch ( Table 3). The three different CNN backbones we tested are vanilla CNN [17], residual convolution blocks [34], and densely connected convolution blocks [19].
The results ( Figure 6

Discussion
Semantic segmentation is an essential task in medical imaging applications; accurate delineation of tumors enhances the outcome of diagnosis, treatment planning, and image-guided surgery. While CNN-based models capture long-range dependencies by inefficiently stacking convolution layers, attention-based models explicitly build relations of all ranges. However, assigning learnable parameters to all pairs of global relations is costly; thus, efficient approximations of global self-attention have been studied to circumvent the inherent quadratic complexity. The most popular technique is to partition the whole image into patches, which are subsequently used to compute (incomplete) global attention across patches [24] or pixel-wise global attention within patches [1]. This arbitrary partitioning mechanism imposes serious problems on existing medical segmentation methods since it cannot capture all attention of different scales and ranges. When large objects are partitioned into different patches, these fragments cannot effectively attend to each other. In this paper, we designed and developed a pyramid deep-learning architecture to address this critical issue.
The proposed method, Pyramid Medical Transformer (PMTrans), implements a multi-scale model architecture and exploits the feature extraction power of both self-attention and convolution layers.
The new model uses multiple input resolutions to comprehend relations of different ranges without changing the overall complexity of self-attention computation. It learns longer-range dependencies on lower resolution images and refined segmentation details on full resolution images.
PMTrans also implements an adaptive partitioning mechanism to reserve informative relations that the existing rigid partitioning mechanism misses. These retained relations, which may be lost during the patch partitioning step in the existing methods, improve the segmentation outcome.
The results of PMTrans on three medical imaging datasets (GLAS [10], MoNuSeg [11], and HECKTOR [12]) showed its superior performance over the state-of-the-art convolution-based and transformer-based models on medical images. PMTrans is a general segmentation method designed for medical images but readily applicable to images from other domains.