{'fraction_non_alphanumeric': 0.04132385938668661, 'fraction_numerical': 0.016828721017202692, 'mean_word_length': 4.877472527472528, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising

Low dose computed tomography is a mainstream for clinical applications. How-ever, compared to normal dose CT, in the low dose CT (LDCT) images, there are stronger noise and more artifacts which are obstacles for practical applications. In the last few years, convolution-based end-to-end deep learning methods have been widely used for LDCT image denoising. Recently, transformer has shown superior performance over convolution with more feature interactions. Yet its ap-plications in LDCT denoising have not been fully cultivated. Here, we propose a convolution-free T2T vision transformer-based Encoder-decoder Dilation net-work (TED-net) to enrich the family of LDCT denoising algorithms. The model is free of convolution blocks and consists of a symmetric encoder-decoder block with sole transformer. Our model is evaluated on the AAPM-Mayo clinic LDCT Grand Challenge dataset, and results show outperformance over the state-of-the-art denoising methods.


Introduction
In recent years, low dose computed tomography (LDCT) has become the mainstream in the clinical applications of medical imaging. However, the low quality of LDCT image has always been a barrier since it compromises the diagnosis value. To overcome this issue, traditional methods (e.g. iterative methods) manage to suppress the artifact and noise by using the physical model and/or prior information. For example, Compressive Sensing (CS) has been widely used for ill-posed inverse problems by learning sparse representations [1], and the representative total variation (TV)-based models assume that the clean image is piecewise constant and its gradient transform is sparse [2][3][4][5]. Xu et al combined dictionary learning and statistic IR (SIR) [6] for LDCT denoising.
Tan et al proposed a tensor-based dictionary learning model for spectral and dynamic CT [7]. Ma et al designed a Non-Local Mean (NLM) method to utilize the redundancy of information across the whole image rather than local operations on neighboring image voxels [8]. Nonetheless, none of these algorithms is adopted in commercial scanners because of the hardware limitations and high computational cost [9].
In last few years, deep learning-based methods attracted more attention for CT denoising and achieved the state-of-the-art performance [10]. Chen et al combined the auto-encoder, deconvolution network, and shortcut connections into a residual encoderdecoder convolution neural network (CNN) for CT imaging [11]. Yang et al used WGAN-VGG in the denoising base network and adopted perceptual loss to evaluate the reconstructed image quality [12]. Fan et al constructed a quadratic neuron-based autoencoder with more robustness and utility for model efficiency in contradiction of other CT denoising methods [13]. However, these convolution-based methods has limited ability to capture contextual information with long spatial dependence in image or feature maps.
Very recently, transformer [14] has gradually become the dominant method in the natural language processing (NLP) [15][16][17][18] and computer vision (CV) fields [19][20][21][22][23][24][25][26][27][28][29][30][31]. Transformer has achieved a great performance in high level tasks, such as classification, object detection, image segmentation, etc. Dosovitskiy et al first proposed vision transformer (ViT) in the CV field by mapping an image into 16×16 sequence words [23]. To overcome the simple tokenization in ViT, Yuan et al further proposed a Token-to-Token method to enrich the tokenization process [29]. Moreover, Liu et al designed a swin transformer to include patch fusion and cyclic shift to enlarge the perception of contextual information in tokens [27]. Researches also explored the transformer for low vision task [21,24,25]. However, transformer in LDCT denoising has not been well explored. Zhang et al designed a TransCT-net to utilize transformer in the high frequency (HF) and low frequency (LF) composite inference [32]. Nevertheless, Zhang's work includes a lot of convolutions in the HF/LF tokenization and Piecewise Reconstruction module. So far, there is no convolution-free model for LDCT denoising. In this paper, for the first time, we propose a convolution-free Transformer Encoder-decoder Dilation (TED-net) model and evaluate its performance compared with other state-of-the-art models.
The rest of this paper is organized as follows. Part II introduces the key methods used in our proposed model. Part III reports the experiment details and results. Part IV discusses some related issues and make a conclusion.

Methods
In this paper, we propose a convolution-free T2T vision transformer-based Encoderdecoder Dilation network (TED-net). As shown in Fig. 1, in the encode part, the model includes Tokenization block, Transformer Block (TB), Cyclic Shift Block (CSB), Token-to-Token block with Dilation (T2TD) and without dilation (T2T). The decoder part includes T2T, T2TD, Inverse Cyclic Shift Block (ICSB) and Detokenization Block.

Noise Model
The LDCT denoising task attempts to recover a clean NDCT image (1) In this paper, we propose a Transformer-based model : N N N N T   to learn the deep features and capture the noise pattern of the image. Then we recover the clean image by combining both the output residual image and the original noisy image. (2)

Transformer Block
In the Transformer Block (TB), we utilize a traditional transformer in the encoder and decoder stage between two T2T blocks which contain Multiple head Self-Attention (3) Fig. 2. The architecture of T2TD block which includes reshape and unfold with dilation.

Token-to-token Dilation Block
Token-to-Token (T2T) block is recently utilized to overcome the simple tokenization of image in vision transformer. Traditional tokenization only includes one tokenization process using either reshape or convolution, while the T2T block adopts a cascade tokenization procedure. We further use dilation in the tokenization process to refine the contextual information fusion and seek relation across larger regions. Fig. 2 illustrates the structure of T2TD block which consists of reshape and soft split with dilation. . Through this operation, the number of tokens are reduced by combining several neighboring tokens into one unit though the embedding dimension is increased accordingly with several tokens concatenated together.

Reshape. Given tokens
As demonstrated in Fig. 2, dilation is also used in the unfold process to capture the contextual information with longer dependence. After the soft split with dilation, the input feature maps n after the soft split operation is calculated as: where , , and dilation kernel stride are related parameters in the Unfold operation.
Cyclic Shift. After the reshape process in the encoder network, we employ the cyclicshift to modify the shaped feature map. The pixel values in the feature map are assembled in a different way that will add more information integration in the model. Then, an inverse cyclic shift is performed in the symmetric decoder network to avoid any pixel shifts in the final denoising results.

Experiments and Results
In this part, the data preparation, experiment settings and comparison results are presented. Our model is trained and evaluated on a public dataset, and the results show that our model outperforms other state-of-the-art models.

Experiment Settings
The experiments are running on Ubuntu 18.04.5 LTS, with Intel(R) Core (TM) i9-9920X CPU @ 3.50GHz using PyTorch 1.5.0 and CUDA 10.2.0. The models are trained with 2 NVIDIA 2080TI 11G GPUs. Here are the details of our experiment setting: in the encoder block, our model consists of three soft split stages, two transformer layers, and two cyclic shift layers, while in the decoder block it includes three inverse soft split stages with fold operations, two transformer blocks and two corresponding inverse cyclic shift operations. One additional transformer layer is between the encoder and decoder parts to further incorporate more feature inference. The kernel size for the three unfold/fold operations are 7×7, 3×3, 3×3 with a stride of (2,1,1) and a dilation of (1,2,1), respectively. Moreover, the token dimension in the encoder/decoder part is 256, and pixel quantity is 2 for the two cyclic shift layers. We use a patch number of 4 for   training and the epochs are 4000. Adam is adopted to minimize the MSE loss with an initial learning rate of 1e-5. In the evaluation stage, we segment the 512×512 image into overlapped 64×64 patches, and only crop out the center part of the model output to aggregate to the final whole predictions to overcome the boundary artifacts.

Comparison results
SSIM and RMSE are adopted to quantitatively measure the quality of the denoised image. Our model is compared with state-of-the-art baseline algorithms: RED-CNN [11], WGAN-VGG [12], MAP-NN [33], and AD-NET [34]. RED-CNN, MAP-NN, and WGAN-VGG are popular low dose CT denoising models while AD-NET has high performance on gray image denoising. We retrain AD-NET on AAPM dataset with the same setting as other methods and obtain a comparison result. Fig. 4 shows the results of different networks on L506 with lesion No.575. Fig. 5 demonstrates the amplified ROIs from the rectangular area marked in Fig. 4. Fig. 6 illustrates the performance of our TED-net on lesion No.576. Fig. 4, Fig. 5 and Fig. 6 show that our TED-net has a better performance in removing the noise/artifact and maintaining high-level spatial smoothness while keeping the details of the target image. However, other methods have more blotchy noisy textures. Additionally, quantitative results from Table 1 also confirm that our model outperforms other models.

Conclusion
In this paper, a novel pure transformer-based convolution-free LDCT denoising algorithm is developed for clinical applications. In contrast, the most state-of-art models are based on CNN. To the best of our knowledge, this is the first research to apply pure transformer for LDCT denoising. Our contributions are mainly three-folds: (1) A convolution-free U-net like T2T-vit-based denoising transformer model is developed. (2) The dilation is used in the T2T stage to enlarge the receptive field to obtain more contextual information from the feature-maps. (3) A cyclic shift is used to furthermore refine the mode of image tokenization. Experimental results show our model outperforms other state-of-the-art models with the highest SSIM value and smallest RMSE value. In the future, this model can be further slimmed with a more powerful tokenization without downgrading of images.