{'fraction_non_alphanumeric': 0.03170415847282843, 'fraction_numerical': 0.014245287543767087, 'mean_word_length': 5.187713310580205, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Vision Transformers in Medical Imaging: A Review

Transformer, a model comprising attention-based encoder-decoder architecture, have gained prevalence in the field of natural language processing (NLP) and recently influenced the computer vision (CV) space. The similarities between computer vision and medical imaging, reviewed the question among researchers if the impact of transformers on computer vision be translated to medical imaging? In this paper, we attempt to provide a comprehensive and recent review on the application of transformers in medical imaging by; describing the transformer model comparing it with a diversity of convolutional neural networks (CNNs), detailing the transformer based approaches for medical image classification, segmentation, registration and reconstruction with a focus on the image modality, comparing the performance of state-of-the-art transformer architectures to best performing CNNs on standard medical datasets.


Introduction
The transformer (Vaswani et al., 2017), identified by its attention mechanism, has become the dominant deep learning architecture in the field of natural language processing (NLP) due to its success in text to speech translation , natural language generation (Topal et al., 2021), text synthesis  and speech recognition (Feng et al., 2022). The success of transformers in the field of natural language processing owes primarily to their ability to capture long range dependencies that aids in the retention of contextual information (Vaswani et al., 2017) as opposed to recurrent neural networks (RNNs) (Graves et al., 2013;Sak et al., 2014) that utilizes a sequential inference process and cannot efficiently capture long range dependencies. A plethora of transformer architectures for natural language processing have been proposed since 2017, a few of the popular architectures are; Bidirectional Encoder Representation from Transformer (BERT) and its variants (Devlin et al., 2019;Lan et al., 2019;, Generative Pre-Trained Transformer (GPT) and its variants (Hoppe & Toussaint, 2020;Radford et al., 2018;Winata et al., 2021).
In the computer vision field, convolutional neural networks (CNNs) have achieved efficient performance mainly due to the structure of their architectures (He et al., 2016;Louis, 2013;Peng et al., 2021;Yang et al., 2022;. Recent evidence show CNNs exploits the locality of pixels aiding capture of vision semantics and yield acceptable performance even on small datasets (d'Ascoli et al., 2021), CNNs are also known to possess progressively enlarge receptive field that aids in the representation of image hierarchical structure in form of semantics. However, the advent of transformers apprised researchers of CNNs' major drawback, the inability to capture long range dependencies such as the extraction of contextual information and the non-local correlation of objects . This has led to attempts to incorporate self-attention either spatially (Cao et al., 2019;Huang et al., 2019;Xiaolong Wang et al., 2018) or channel-wise Woo et al., 2018), into the conventional CNN architecture. Eventually, the first pure transformer for computer vision application named Vision Transformer (ViT) was proposed by (Dosovitskiy et al., 2020), in which they demonstrated the equivalence between multi-head self-attention attached to a multi-layer perceptron and CNNs by considering image classification as a sequence prediction task hence utilizing patch down-sampling and quadratic positional encoding in order to capture long-range dependencies between image tokens (patches). In recent literatures, these pure or hybrid vision transformers (ViTs) have achieved state-of-the-art performance over the CNN benchmarks. This has been consistent across a is variety of computer vision tasks, such as image classification (Dosovitskiy et al., 2020), image reconstruction (Jiang et al., 2021), pixel segmentation ( Zheng et al., 2021), image captioning (Cheng et al., 2021), three-dimensional imaging (Zhou et al., 2021) and video applications (Zhou et al., 2018).
CNNs have substantially influenced the field of medical imaging because of the pertinent need for classification, segmentation and detection, required of the variety of imaging modalities including ultrasound, X-ray radiography, magnetic resonance imaging (MRI), computed tomography (CT), wholeslide-images (WSIs) etc. (Darby et al., 2012). Surprisingly, about 90% of all healthcare data are compiled instances of the various medical imaging modalities. This implies that there is an outlay of data available to foster efficient modelling for clinical diagnosis and decision-making. CNNs (He et al., 2016;Hu et al., 2020;Weng & Zhu, 2021) have excelled because of the ability to learn spatiotemporal dependencies within an image and utilize this in the extraction of distinguishable representation Susanti et al., 2017;Yu & Helwig, 2022). However, convolutional layers have stationary weights that do not adapt for a specific input image, offer their models a limited effective receptive field that limits the ability to capture long-range dependencies between pixels.
The successes of transformers on natural images, have encouraged researchers to query further the application of self-attention in medical imaging in order to effect long-range dependencies between pixels. Additionally, transformers have achieved comparable performance to state-of-the-art CNNs on medical image classification , detection (Ghaderzadeh & Asadi, 2021), segmentation (Tragakis et al., 2022) and reconstruction . Literatures have recorded transformers of better performance than state-of-the-art CNNs however the performance of transformers over CNNs is still debatable and newer modifications to the transformer architecture emerge every day in an attempt to mitigate transformer related problems. This paper attempts to provide a detailed review on the application of transformers in medical imaging and to compare their performance with state-ofthe-art CNNs.
The paper provides detailed and recent review on of transformer state-of-the-art in medical imaging and detailed comparison between CNN and transformer benchmarks. To aid in visual identification and comprehension we have included a taxonomy articulating the disparate application of transformers in medical imaging with references.
Our paper is outlined as follows: the preliminaries of the original transformer, and CNNs by detailing the various combinations with transformers as can be found in the literature. The current progress of the transformer state-of-the-art in medical image classification, segmentation, registration, and reconstruction. In addition, the paper, identifies all known positives and negatives of the transformer, and provide a detailed comparison between present state-of-the-art transformer approaches and that of CNNs.

Attention in Transformers
The fundamental transformer architecture as proposed by Vaswani et al. (2017) is a sequence-tosequence model that is composed of self-attention and point-wise feed-forward network (FFN) block that extracts global dependencies between tokens Self-attention was designed to emphasize relationships between data regardless of their position in the sequence. It is mathematically expressed by a map function of queries, keys and values such that for each input ∈ ℝ , = 1, … , there exist a query ∈ ℝ × , a key ∈ ℝ × , and a value ∈ ℝ × , which are utilized in generating learning parameters , , respectively.
The output is a probability that requires normalization, usually achieved by a softmax function to attain an output distribution represented by the equation (2).
The output of a self-attention block is the sum of element V and the matrix A equipping this attention block with the ability to capture global dependencies within a data.
Multi-head self-attention can be applied to better capture hierarchical features. These are computed in parallel after the final output is obtained by concatenating each individual attention block. This operation is similar to the use of multiple kernels within convolution operations, mathematically expressed by: Here ℎ represents the total number of heads and represents an output matrix of the concatenated projection of all selfattention , , of the ℎ attention.

Point-Wise Feed-Forward Network
The output from the multi-head self-attention block is fed into a feed-forward network comprising two linear activation function and a rectified linear unit (RELU) activation, as expressed in equation (4).
Here represents the output from the previous layer and , , , are trainable parameters of dimensions and , represented as ∈ ℝ × and ∈ ℝ × where = , . It is to be noted that should always be larger than .

Positional Encoding
Learnable parameters are typically employed to aid the network retain positional information, this could be achieved by recurrence or convolutions however, in the transformer architecture this is achieved by inputting information about the position of tokens (words or patches) into the sequence. Vaswani et al. (2017) experimentally utilized sine and cosine functions of varying frequencies Here represents the initial position of the vector while represents the length of the vector, represents the particular instance. In the first pure vision transformer by (Dosovitskiy et al., 2020) the learned position is outputted as a vector and serves as input into the encoder. This vector is a sequence of n-dimension patches where = 1, … , . While a lower n-value will store better positional information; it will also be computationally expensive.

Vision Transformers
A variety of transformer based vision models exists, a few of the prominent ones are; Vision Transformer (ViT) (Dosovitskiy et al., 2020), Detection Transformer (DETR) (Carion et al., 2020), dataefficient image transformer (DeiT) (Touvron et al., 2020) and Swin-Transformer (Ze Liu et al., 2021). The ViT was the first pure adaptation of the vanilla transformer in the field of computer vision. It entails an encoder and a task specific decoder structure where input images are broken into a sequence of non-overlapping patches of size ( × × ). Here C represents the number of channel of the image and P represents its length and width. The information about the position of patches in the image is converted into a vector. This positional information, along with the sequence of non-overlapping patches are fed into the encoder block of the transformer containing multi-head selfattention, layer normalization and a multi-layer perceptron (FFN), this is depicted in Figure (2).
DETR is a hybrid that attaches a transformer encoder to a CNN architecture; it was the first attempt at partially conveying attention from the vanilla transformer to a vision task of object detection. The swin-transformer was designed to reduce the computational cost of the ViT and was tested majorly on segmentation tasks. Liu et al., (2021) attempted varying patch sizes so as to reduce the requirement of full attention due to the input sequence of non-overlapping patches; they introduced the shifted window attention that creates patches of various sizes in a hierarchical order, and further assists in preserving spatial information.
The efficiency of the ViT could only be observed during large scale training as it performs poorly on small datasets, and Touvron et al., (2020) proposed the DeiT in an attempt at solving this problem. DeiT adopts a CNNs teacher and a transformer student structure in a knowledge distillation framework in which a distillation token is added for the purpose of learning from the teacher model. This knowledge is then inherited by the student model while imparting inductive bias.

Hybrids
Hybrids including DETR and DeiT can be grouped into three categories, according to the work of Jun . These categories are; ConvNet-like-Transformer, Transformer-like-ConvNets and Transformer-ConvNet hybrids.
ConvNet-like-Transformers are vision transformers that inherit the properties of conventional CNNs with the aim of improving the efficiency of the transformer. The DeiT (Touvron et al., 2020) is an example of this type of transformer, it attempts to develop transformers that inherit the inductive bias present in convolutional neural networks. Other examples include; Swin Transformer (Ze Liu et al., 2021), HaloNets , DAT (Xia et al., 2022) and PVT . Transformerlike-ConvNets are convolutional neural networks that inherit some of the properties of transformers by sparing or partial integration of transformers into the architecture. The most prominent properties researchers try to integrate to CNNs is self-attention from transformers. Few example include; CoT (Yehao Li et al., 2022), BoTNet  and ConvNext (Zhuang Liu et al., 2022). Transformer-ConvNet hybrids try to form architectures that consist of convolutions, multilayer perceptrons and multi-head self-attention blocks in an attempt to fully leverage the strength of both architectures and form models that are more efficient. A few examples include CvT , Mobile-former , Conformer (Peng et al., 2021), CoAtNet  and ConViT (d'Ascoli et al., 2021).

Datasets
Transformers are generally known to perform better in large training than small sized training due to the absence of inductive bias that bolsters few shot learning. In an attempt to solve this problem, researchers have proposed several hybrid architectures that seeks to incorporate strengths of the convolutional neural network into the transformer. The availability of public medical datasets of diverse modalities have been a major deterrent to the training and re-training of state-ofthe-art CNN architectures like ResNet (K. He et al., 2016) and EfficientNet (Tan & Le, 2019) for the development of domain specific weights to serve as a feature extractor layer in transfer learning for both CNNs and Transformers. A few researchers have proven that transformers benefit more from transfer learning than CNNs (Caron et al., 2021;Raghu et al., 2019Raghu et al., , 2021. However, standard weights like ImageNet do not serve as efficient feature extractors for medical imaging tasks across architectures (Hosseinzadeh Taher et al., 2021) hence the need for a detailed outline of publicly available medical datasets. The detailed compilation of most of the publicly available datasets of various medical image modalities with their description; their download link is provided in within their publication in Table 1 (Parvaiz et al., 2022).

Modality Dataset Description Histopathology Images
Cancer Genome Atlas (Weinstein et al., 2013) The dataset represents heat maps of 33 tumor types, and 3 distinct expressions; reverse-phase protein array, gene expression and miRNA.

Dataset Description Histopathological Images
Cancer Histology Dataset (Kather et al., 2016) These is a dataset comprising whole-slide-images of colorectal cancer cells.
COVID-CT (X. Yang et al., 2020) A dataset containing a number of chest scans complied from a number of literature. CT-emphysema-DB (Sørensen et al., 2010) A dataset of 115 high-resolution scans and 168 manually annotated square patches. LUNA16 (Setio et al., 2017) A dataset containing lung scans for normal and lung nodule classification. LIDC-IDRI (Armato et al., 2011) A large dataset containing lung scan for the diagnosis and screening of thoracic cancer.
BIMCV-COV19+ (Vayá et al., 2020) This is a database of chest x-rays and CT images.
Extensive-XR-CT (X. Yang et al., 2020) A dataset of CT and X-rays for patients with and without covid.

Posterior-Anterior
Chest Radiography COV19 (Haghanifar et al., 2022) This is a combination dataset of about 15 smaller chest X-ray datasets.

Modality
Dataset Description CT-scans IMDTD-18  A dataset containing about 9000 OCT scans.
Kits19 (Heller et al., 2022) A dataset annotated for the segmentation of renal tumor.
CHAOS (Kavur et al., 2021) An MRI annotated dataset for abdominal organ segmentation: kidney and liver. UKBB (Sudlow et al., 2015) A dataset for identifying the causes of a wide range of complex diseases. BrATS-20 (Bakas et al., 2017) A dataset for brain tissue segmentation, with about 2000 images. Iseg-17 (Li Wang et al., 2019) A dataset for brain tissue segmentation consisting of 20 images.
DICRLN (Shiraishi et al., 2000) A database of annotated chest images for detecting lung nodule. IN-breast (Moreira et al., 2012) A database containing annotated breast mammograms.

Classification and Segmentation
Classification of medical images is a vital task in healthcare because of the increasing need for diagnosis, identification and distinction of healthcare images and relevance in intended applications. Varieties of transformer architectures have been employed in modelling with the aim of achieving higher efficiency than previously obtained. Researchers have employed pure vision transformers an d hybrid models of disparate architectures for classification of medical images in literature, hence the extensive review of the various transformerbased methods developed for the classification of medical images, with a focus on the image modality, as presented in the writeup.

Histopathological Images
Various staining substances and methods have aided the diagnosis, detection and classification of tumors and carcinomas in pathology. These slide images, in digital form, serves as a resource bank for the development of computational models to perform automatic diagnosis, to facilitate efficient and faster diagnosis. The earliest known ViT utilized for histopathological image classification, in literature is TransMIL , which integrate transformers to a multiple instance learning (MIL) framework, in order to introduce correlation between various instances. In this work, MIL is achieved by pooling operations performed on learning instances extracted by a pre-trained CNN (He et al., 2016). These instances, after squaring, are passed into a block containing multi-head selfattention (MSA), positional encoding and an MLP for weakly supervised classification. The resultant models achieved AUC scores of 93.09% and 96.03% in classifying binary tumors and cancer subtypes respectively. Wang et al. (2022) introduced a contrastive learning strategy based on self-supervised learning (SSL). In general, SSL is known to ameliorate the annotation of large-scale data by utilizing an unlabeled dataset in the extraction of useful representations that generalizes well to multiple downstream tasks. Semantically relevant contrastive learning (SRCL), unlike traditional contrastive learning, seeks to acquire more representations that are informative by aligning closely related positive pairs. This is achieved by pre-training CTransPath (an integration of CNNs and multiple swin transformers) with a plethora of unlabeled data, to serve as a feature extractor for the downstream task performed by SRCL. GasHis-Transformer  introduces an approach for gastric histopathological image classification based on global and local information capture. Local information capture is realized by a CNN architecture while global information capture is accomplished by attention blocks coupled to CNNs. Advocates of selfsupervised ViTs in histopathology  proposed a novel vision transformer based on the Hierarchical nature of WSIs. Hierarchical image pyramid transformer (HIPT) , employs a two-level self-supervised learning framework to exploit both the hierarchical nature of WSIs and the large sequence length of WSI tokens as a result of their pixel nature. This is performed by aggregating visual tokens at three levels (cell, patch and region) in order to form the slide representation and utilizing self-attention as a permutationequivariant aggregation layer; the HIPT with hierarchical pre-training yielded better performance than SOTA methods for survival prediction and cancer subtyping.

Computed Tomography Scans
CT-scans can create detailed cross-sections of bones, soft tissues, organs and blood vessels, these sections can also be formatted to multiple frames that aid in the generation of three-dimensional images. The application of ViTs in computational tomography has largely been focused on thoracic diseases because of the contrast between gas and tissue. These ViTs are designed for two-dimensional, as well as three-dimensional CT images. CTNet as proposed by Liang et al. (Liang et al., 2021) is a hybrid model comprising a CNN feature extractor and ViT for the detection of COVID-19 from threedimensional chest scans. After training on the COV19-CT dataset, it achieved 88% in F1 evaluation. Barhoumi et al. (2021) proposed scopeformer, a hybrid model for the classification of intracranial hemorrhage from CT images. Their work demonstrates that by stacking several Xception CNN blocks and aggregating their feature maps, a feature-rich map can be developed to serve as the input to a ViT and improve the model performance. Consequently, the model achieved a test accuracy of 98.04% on the corresponding classification task.
The effective diagnosis of pancreatic cancer from two-dimensional CT images was demonstrated by Xia et al (2021). In this work, a region-of-interest (ROI) feature map is developed from annotated training data based on the U-net segmentation algorithm. This pancreas ROI feature map is fed as input to a transformer that is built upon the U-net algorithm for segmentation. The model is trained on a 3-class labeled data for classification, and achieved specificity and sensitivity of 95.2% and 95.8% respectively after training on a dataset of 1321 patents. Swin UNETR (Tang et al., 2021) is a prominent model for the classification of 3 dimensional CT images. It implements a hierarchical encoder for self-supervised pre-training, it was finetuned for classification and segmentation tasks at the Beyond the Cranial Vault (BTCV) challenge. Their model outperformed all other models submitted for the challenge.  M3T (Jang & Hwang, n.d.) Pure ViT MRI Brain 2D Accuracy The work pre-trains a ViT on ImageNet and fine-tunes it for brain tumor classification.
Tummala et al. (Tummala, 2022) Pure Matsoukas et al. (Matsoukas et al., 2021) The segmentation of CT-scans is important in the diagnosis, evaluation and monitoring of various phenomena in healthcare. Swin UNETR (Tang et al., 2021) also performs 3D segmentation primarily for computed tomography images. It attaches selfsupervised heads, aggregated by contrastive learning strategy to a swin transformer encoder. It also achieved state-of-the-art performance in segmentation. Similarly Xia et al (2021) hybrid model train by classification supervision can be utilized for CT segmentation tasks as well, and for CT classification task as discussed above. Because of the computational cost involved in modelling global representation on full resolution images  proposed PMTrans, a hybrid model that incorporates mulit-scale attention to a CNN feature extractor with the aim of capturing diverse range relationships from multi-resolution images. This method is recorded to have outperformed other CNN and Transformer based models at the time of publication. UNETR  was designed for semantic segmentation of 3D brain MRI and spleen CT scans. It attaches a transformer encoder to a Unet-like CNN decoder for semantic segmentation.

Magnetic Resonance Imaging
MRI provide very detailed anatomical images due to its powerful and effective non-invasive imaging technology: it also produces two-dimensional as well as three-dimensional images. It is therefore of importance to consider the dimensionality of the image instance that the modelling method is suited.
M3T ( He et al. (2022) and Shen et al. (2022). On the one hand, the former attempts at estimating brain age by a global-pathway-local-pathway transformer network that aids the simultaneous capture of global and local representations from images. The particular model was trained on a collective of six publicly available dataset. After evaluation on two additional datasets, it exhibited state-of-the-art performance in age prediction. On the other hand, the latter attempted at extracting age-specific morphological information from Fetal MRI to promote age-based classification. They also demonstrate the inferiority of traditional CNN approaches to that of attention-guided CNN methods: achieved by incorporating attentionguided mask inference to a traditional ResNet-50 architecture. Their method resulted in R 2 score of 0.945 and MAE of 6.7 days.
The fully convolutional transformer (Tragakis et al., 2022) leverages the success of U-net algorithm in development of a fully-convolutional, depth-wise transformer that replaces the convolution blocks in U-net architecture with fully convolutional transformers. This method aids efficient segmentation that considers the fine-grained nature of medical images. Their model extracts global information as well as capturing hierarchical attributes from features. UNetFormer  attempts to adapt the dubbed UNetFormer architecture by redefining each encoder block with swin transformers and each decoder block with CNNs; they also included skip connections at different data resolutions. For encoder pre-training, they incorporated self-supervised learning with the aim of letting the model randomly predict masked volumetric tokens.  Another modelling approach is to convert threedimensional images to two-dimensional slices (Jun et al., 2021), utilizing only two-dimensional convolutions. This network is pre-trained using DINO self-supervision, after which the twodimensional slices are re-combined for the purpose of prediction. This method has been successfully applied for regression and classification, as well as segmentation tasks.

X-ray
The passage of X-rays through a body can aid the generation of informative images of tissues and the internal structure of the body. X-rays are inexpensive and convenient and have a wide range of application in medical imaging such as cancers, fractures and various pneumonia (including COVID-19) etc. Thus, they are an important source of medical data capture to clinicians and researchers alike. SEViT (Almalik et al., 2022) attempts to model self-ensemble transformers based on experiment that proved that the feature representations learned by the initial ViT blocks are generally unaffected by adversarial perturbations. In order to model resilience to adversarial attacks, they proposed learning multiple classifiers that aggregates feature representations learned from initial ViT blocks to those learned from final ViT blocks. This method leverage the details presented from the final ViT representation as well as the robustness of intermediate ViTs. Mondal et al (2022) employed a pure transformer for the detection and classification of COVID-19 from X-ray images. In their work, a multi-stage transfer learning strategy based on ViT is adopted where a ViT is pretrained on domain specific data for the extraction of domain relevant information. This feature extractor is attached to a conventional transformer that performs classifications based on its fully connected multi-layer perceptron (MLP). Chest X-ray data and other X-ray forms required for adequate modelling are scarce. However, availability of such data is on the rise. With an increase in the available data comes increasing demand for annotation.  tried to provide solutions to these issues in his works p-FESTA  and DISTL . For scarcity of training samples, they proposed p-FESTA that employs multi-task distributed learning that is based on federation and shared learning. They achieved this by exchanging the CNN in the original Federated Split Task Agnostic (FESTA) with random patch permutation with an aim of improving performance of the multi-task learning while maintaining privacy. For the problem of annotating newer datasets, they proposed DISTL that incorporates self-supervision and self-training to ViTs with the aim of developing a model that can learn useful representations from un-annotated data. DISTL was evaluated on data from three hospitals and improved in performance with increase in the size of unlabeled training data. CoTr (Xie, Zhang, Shen, et al., 2021) Hybrid MRI Brain 0|0|1 3D Dice Converts a 3D image into twodimensional slices, processes image in 2D then recombines as output.
Medical Transformer (Jun et al., 2021) Pure ViT X-ray, CT Lung 1|1|1 2D IoU, Accuracy Utilizes several swin-transformers with patch and positional encoding to achieve classification and segmentation training.
Sun et al. (Sun & Pang, n.d  A lot of the research on the X-ray modality is performed for classification however, there are a few works that attempts the segmentation of chest x-rays. Sun et al. (Sun & Pang, n.d.) Developed a model for the classification and segmentation of chest X-rays based solely on the swin transformer. So far, very few models employ just the transformer architecture in performing segmentation tasks. Yet they achieved a segmentation accuracy of 95% from three variations of their swin-transformer model. The p-FESTA method discussed earlier also tested for segmentation of X-rays; inducing federation and shared learning towards the performance of segmentation tasks after self-supervised pretraining. TransAttUnet  was developed for the accurate segmentation of organs and lesions from X-ray and CT imaging by defining an encoder-decoder network with multi-scale skip connection for processing higher resolution images and multi-level guided attention for mapping global relationships between multiple resolutions. These skip connections were also applied to the decoder part of the architecture to aggregate semantic-scale up-sampling features, alleviating information recession and developing a more detailed pixel map.
To aid in root canal therapy assessment  developed GT U-Net that retains the depthwise nature of U-net but replaces the convolutions with group transformer-hybrids. The idea behind implementing group transformers rather than individual vanilla-type transformer is to reduce computational cost. They also defined a shapesensitive Fourier Descriptor loss function in order to aid model optimization.

Fundus
Fundus photography aids in the diagnosis of a variety of medical conditions. Due to its complex nature; comprising retina, macula, optic disc, fovea and blood vessels, it is expedient that computation be employed in more efficient and reproducible diagnostics. Rui et al. (2021) developed an encoderdecoder decoder structure with a pixel relation based encoder and a filter based decoder. This model employs weakly supervised training via filter based transformer decoder, as well as lesion region importance and lesion region diversity to enable the model learn filters well. The model was tested for DR grading and lesion discovery, and recorded stateof-the-art performance.
With the aim of aiding ophthalmologist in the automatic segmentation of diabetic retinopathy lesions Huang et al (2022) developed RT-Net following a clinical approach: They investigated the pathogenic causes of diabetic retinopathy lesions and found that certain lesions present relative patterns with each other and appear close to specific vessels. This finding aided the proposition of a relation transformer block composed of selfattention for global relationship and cross-attention that enables interactions between lesion and vessel features. They also proposed a global transformer block to aid the capture of finer details of small lesion patterns. Their dual-transformer approach was capable of segmenting four kinds of DR lesions and attains state-of-the-art performance. Segtran  was developed with the aim of capturing fine details as well as global features simultaneously. They successfully developed a transformer-based approach that has unlimited effective receptive field at high and low resolutions by utilizing squeeze and expansion transformer networks, with each transformer block performing a unique function: the squeeze block attempts regularization of the attention block to aid global information capture, the expansion block learns a diverse array of representations. Tested on the BraTS dataset (Bakas et al., 2017) the model achieved the most segmentation accuracy.

Archi Modality Organ Type Evaluation Reference
Hybrid MRI Brain 3D Proposed slice-to-volume registration that predicts the transformation of a slide based on information from other slides.
SVoRT (Junshen Xu et al., 2022) ViT MRI Brain 3D Utilizes a pure transformer for the projection of surface data on a curved manifold SiT (Dahan et al., 2022) Hybrid MRI, CT Multiple 3D Presents a hybrid transformer for image registration along with 2 model variants.
TransMorph (J.  Hybrid CT Multiple 3D An encoder-decoder architecture composed of transformer blocks for denoising medical images. Eformer (Luthra et al., 2021) ViT CT Multiple 3D Comprises a symmetrical encoderdecoder architecture solely based on transformers for denoising. MIST-net

Registration and Reconstruction
Medical image registration becomes relevant when we intend to analyze the same image captured with different modalities or at different times. Registration aims at establishing relationships between static and moving images by finding dense per-voxel displacement. In recent times, transformers are seen as the architecture of choice for extracting relating features from multimodal images required in registration tasks because of they offer better understanding of spatial representations. SVoRT (Junshen  attempts volume registration from slides based on an attention mechanism. Features extracted by a ResNet architecture, as well as the position of slices in 2D and 3D are fed into the transformer to encode spatial and global representations. SVoRT was tested on real world data and achieved state-of-theart performance. Dahan et al (2022) developed a transformer-based technique for the projection of surface features to a curved surface by leveraging multi-head self-attention and spatial encodings.
The transformation of signals into an interpretable image that can serve as data for diagnosis is a task that is performed efficiently by transformers.

Registration and Reconstruction
(1) This progress has led to; the reduction of the number of MRI measurements required to establish a scan, aid in reducing the radiation dose required in CT scans, ease of rebuilding surgical scenes. DSFormer  presents a selfsupervised learning approach based on the transformer and aids the acceleration of multicontrast MRI reconstruction. It achieves this by developing deep conditional cascade transformer from swin transformers with two unique strategies to encourage information sharing. During evaluation, DSFormer achieved almost equivalence performance when train on full supervision and selfsupervision. T 2 Net ) developed a hybrid model of convolutions and transformer architecture for MRI reconstruction and superresolution by situating task transformer networks after two parallel CNNs. Feng et al. (2022) presents (MTrans) that utilizes a transformer architecture for multi-modal MR imaging for the purpose of global information capture. They additionally defined a cross-attention module to extract multi-scale information from the different modalities and recorded state-of-the-art performance upon evaluation. MIST-net  was developed for sparse-view tomographic reconstruction. It entails a robust architecture of three modules; initial recovery based on a flexible network architecture comprising convolution and pooling layers arranged in an encoder-decoder structure; data consistency aimed at preserving edge information by utilizing an enhancement filter defined by convolution and concatenation operations, high-definition reconstruction via swin transformers. This model performed efficiently on evaluation.

DISCUSSION
Transformer architecture and its application cut across various imaging modalities in medical imaging. However, there hasn't been a clear and detailed comparison between transformers and their CNN counterpart. Comparatively, we consider the performance of both architectures; with the exploration of the strengths and weaknesses as well as on their performance under various scenarios.

Key Properties of Transformers
A few distinctive properties govern the behaviors of transformers. A good understanding of its strengths and drawbacks have informed researchers on ways to deploy transformers for the creation of problemspecific and more efficient models that are relevant in the real world. These properties are discussed below.

Long-Range Dependency
In natural language processing (NLP) long-range dependency can be viewed as the preservation of contextual information in a sequence of word tokens, this can be transmitted to computer vision as the relationship between patches of an image (Devlin et al., 2019). This feature can be attributed to the multi-head, self-attention module that maps all token together with a constant distance thereby capturing token-to-token relationships. CNNs lacks such feature hence have limited receptive fields .

Model Capacity
Transformers aggregate projections progressively at a constant scale whereas CNNs aggregate projections through a series of convolution and pooling operations that continuously rescales the image. Constant scale multi-processing aids in better preservation of global information than rescaling operations (Yanghao Li et al., 2022). Similarly, transformers have a better loss landscape than CNNs due to self-attention, and it leads to better generalizability .

Integration and Manipulation
Due to the dynamic nature of transformers, a variety of architectures, hybrid or pure, can be created. This property is necessary for reducing the limitations of the method. Additionally, their performance improves with training size and model capacity, a property that also induces increase computation cost and time.

Adversarial Noises
One disadvantage of CNNs is their vulnerability to adversarial noises, these limit the model's ability to output an ideal representation of the input data (Choi et al., 2022). Transformers are generally more robust to perturbations and corruptions (Bhojanapalli et al., 2021).

Inductive Bias
The scale-adjustment processing employed in convolutional neural networks accords them the ability to extract more local information from individual pixels, leading to faster convergence and better performance when trained on smaller datasets (Cordonnier et al., 2019). This is not the case with transformers because same-scale processing in transformers capture more global, than local, information (Ramachandran et al., 2019).

Transformer and CNNs
Considering the properties of both architectures and optimizations performed to mitigate their limitations, a comparative analysis on how both model types perform on different domains, considering; training from initialized weights, transfer learning, self-supervision and inference.

Training From Randomly Initialized Weights
Random weight initialization requires the model to learn representations from scratch and; only from that which was fed into the model for training. The performance of transformers trained from scratch and CNNs (He et al., 2015) are compared in a variety of literature (Fauw, 2022;Shuang Yu, 2021). On smaller datasets, CNNs seem to have performed better because of the inductive bias inherent within their architecture. However, as the data size increases, transformers are seen to gradually out-perform their CNN counterpart. Training from scratch is seldom practiced in medical imaging due to the small size of most medical data.

ImageNet and Same Domain Pre-training
To augment for the size of most domain specific data in computer vision and medical imaging, ImageNet pre-trained weights are often employed for model initialization: this usually results in improved performance (Morid et al., 2021). Researchers have investigated the benefits and extent of transfer learning to ViTs. They discovered that both CNN and ViTs seem to benefit from ImageNet pre-trained weight and same domain pretraining as well, it is also recorded that transformers benefit the most from transfer learning (Hosseinzadeh Taher et al., 2021;Raghu et al., 2019).

Self-Supervision
Self-supervision is the most effective solution to the problem of lack of large sized, well-annotated data for modeling in computer vison and medical imaging. The most utilized self-supervised learning schemes; BYOL and DINO, have been found to achieve performance that are comparable to supervised learning schemes. This has prompted their utilization, alongside supervised fine-tuning, in computer vision; achieving state-of-the-art performance (Afouras et al., 2020;Hendrycks et al., 2019;Kolesnikov et al., 2019;Jiaolong Xu et al., 2019). This concept have been attempted on a variety of medical image modalities (R. J. Jun et al., 2021;Xiyue Wang et al., 2022; also recording state-of-the-art performances. Additionally, ViTs are found to benefit slightly more from self-supervised pretraining than CNNs.

Inference without Fine-Tuning
In cases of extremely scarce data, features extracted from a pre-trained network can be utilized directly for classification and clustering operations. This is most optimal when the pre-trained features are closely related to the target features. One research assessed whether ViTs performed better that CNNs in inference without fine-tuning by applying k-NN evaluation on the penultimate layer of a CNN and the CLS token of a ViT. They evaluated for both indomain and out-domain pre-trained weights and recorded that ViTs performed better in both scenarios (Fauw, 2022).