{'fraction_non_alphanumeric': 0.029855686482192507, 'fraction_numerical': 0.009665033761419304, 'mean_word_length': 5.0524839743589745, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Attentive Wasserstein Generative Adversarial Networks for MRI Reconstruction with Recurrent Context-Awareness

The performance of traditional compressive sensing-based MRI (CS-MRI) reconstruction is affected by its slow iterative procedure and noise-induced artefacts. Although many deep learning-based CS-MRI methods have been proposed to mitigate the problems of traditional methods, they have not been able to achieve more robust results at higher acceleration factors. Most of the deep learning-based CS-MRI methods still can not fully mine the information from the k-space, which leads to unsatisfactory results in the MRI reconstruction. In this study, we propose a new deep learning-based CS-MRI reconstruction method to fully utilise the relationship among sequential MRI slices by coupling Wasserstein Generative Adversarial Networks (WGAN) with Recurrent Neural Networks. Further development of an attentive unit enables our model to reconstruct more accurate anatomical structures for the MRI data. By experimenting on different MRI datasets, we have demonstrated that our method can not only achieve better results compared to the state-of-the-arts but can also effectively reduce residual noise generated during the reconstruction process.


Introduction
Compressed sensing magnetic resonance imaging (CS-MRI) [1] has been proposed for accelerating MRI process. This technique uses a small fraction of data to reconstruct images from sub-Nyquist sampling. Assuming the raw data is compressible, CS-MRI performs nonlinear optimisations on the undersampling data without sacrificing the quality of the reconstructed images significantly.
However, it is still very challenging to consolidate the speed of reconstruction and robustness of image quality maintenance in one CS-MRI based framework. On the one hand, CS-MRI tries to solve underdetermined equations to perceive the original signals from the limited undersampled data. This requires nonlinear optimisation solvers for a common non-convex system that usually involve iterative computations, which can result in prolonged reconstruction time [2]. On the other hand, CS-MRI may produce images with degraded image quality and low signal-to-noise ratio (SNR) from randomly highly undersampled k -space data [3]. Moreover, in addition to a large amount of computation needed for the nonlinear optimisation, CS-MRI also requires that the acquisition matrix and the sparse transformation matrix are unrelated. Based on the above limitations, the acceleration factor of CS-MRI is generally between 2 and 6.
Recently, deep learning-based CS-MRI methods have emerged as an effective way to solve the problems of slow and unstable MRI reconstruction [4,5,6,7,8,9,10,17,18,19,20,21,22,23,24,25]. For example, a conditional Generative Adversarial Networks-based model (DA-GAN) was proposed to achieve fast CS-MRI [5], but still, this end-to-end training neglected the correlation between adjacent 2D slices. Thus, although DAGAN can achieve fast MRI reconstruction, it may lose image quality without using a priori information. For another example, DC-CNN [6] applied cascades of convolutional neural networks with a residual connection for CS-MRI. Besides, DC-CNN also used a data consistency (DC) step to ensure that the output of each cascade was consistent with the original k -space information. However, DC-CNN approach was not able to effectively utilise the full temporal domain information. In contrast, a convolutional recurrent neural network (CRNN) method was proposed to incorporate a bidirectional convolutional recurrent unit for a faster and more stable reconstruction [7]. However, such an approach was not able to effectively exploit the k -space information from individual images.
In this study, we propose a GAN based architecture that works on continuous sequential data for CS-MRI. This intuitively mimics the way reporting clinicians scrutinise the 3D data by scrolling up and down to fully sense the information above and below the current 2D slice. Our method can not only overcome the shortcomings of slow reconstruction but can also maintain higher reconstructed image quality by combining the characteristics in time and frequency domains.
To the best of our knowledge, this is the first study to combine Recurrent Neural Networks (RNN) with GAN in the field of MRI reconstruction. In particular, we design a novel generator with bidirectional convolutional long short-term memory (Bi-ConvLSTM) that can encode the a priori frequency and time-domain information. Besides, another significant contribution of our work is that we propose a spatial attention-based model that the attention unit in our model can distinguish between significant and non-significant features in terms of the MRI reconstruction task. In addition, we utilise WGAN with gradient penalty (WGAN-GP) as a critic function, which can significantly improve the stability of GAN. We also couple the adversarial loss with pixel-wise mean square error (MSE) and the perceptual loss [11] to achieve better reconstruction details with superior perceptual image quality.

Problem formulation of CS-MRI
Deep Learning-based CS-MRI Reconstruction. Let x ∈ C D represents the slice of 2D images to be reconstructed, which consists of √ N × √ N pixels for one image, and let y denotes the undersampled measurements in k -space. For deep learning-based methods, previous studies such as [4] and [6] incorporated a CNN into CS-MRI reconstruction, transformed the unconstrained optimisation problem into: in which f cnn denotes the forward propagation of data through the CNN parametrised by θ, and ζ is a regularisation parameter. X u is the reconstruction from the zero-filled undersampled k -space measurements.
In the reconstruction network selection, many previous studies, e.g., [5,10,17,18], relied on an encoder-decoder structure. Nevertheless, our preliminary experiments indicated that these single structures performed poorly in the PSNR. Moreover, there are also methods [6,7] that developed for the dynamic MR reconstruction, but they did not perform well at higher k -space undersampling.

DAWGAN for CS-MRI
In this study, we propose a Deep Attentive Wasserstein Generative Adversarial Networks (DAWGAN) method to reconstruct MRI images from highly undersampled data with continuous sequential data. It contains three key components: a Bi-ConvLSTM block, a spatial attention block (SAB) and a WGAN-GP as the critic function. The workflow of our DAWGAN is summarised in Figure 1.
Image Domain Feature Extraction via a Sequential Learning. To achieve more aggressive undersampling, one way is to encode the a priori frequency and time-domain information in sequential data, e.g., 2D MRI slices of a 3D volumetric data. We assumed X as the feature representation of our 2D sequential MRI data slices throughout the 3D volume. Here X (i) l denoted the representation at slice l and iteration i. We needed to take into account X in the reconstruction process to provide information for X (i) l . To that end, we proposed a Bi-ConvLSTM subnetwork to exploit both temporal and iteration dependencies jointly. The Bi-ConvLSTM subnetwork can be formulated as: l,t denoted the backward direction. Through Bi-ConvLSTM layer, our model can learn the differences and correlations of successive MRI data slices. The output of the Bi-ConvLSTM layer then took a refinement connection to prevent data shifting.
Spatial Attention Block (SAB). The main aim of the designed SAB was to increase representation power by using attention mechanism: focusing on important features and suppressing unnecessary ones. Details about the SAB are shown in Figure 1. Inspired by [12], we set the SAB after the first convolution block, which was also propagated to the up-sampling layers with the skip-connections. We conducted average-pooling and max-pooling operations on the feature map obtained from the upper layer to generate an efficient feature descriptor. Then we utilised a convolution layer to generate a feature map that could encode where to emphasize or suppress. The SAB took all the features extracted by the upper layer to calculate the attention map.
We assumed that the 2D maps generated by pooling operations were F avg ∈ R 1×H×W and F max ∈ R 1×H×W . Each denoted average-pooled features and maxpooled features across the feature map. The two maps were then stacked and convolved by a standard convolution layer to produce the 2D spatial attention map. Hence, our spatial attention map was computed as where f 7×7 represented the convolution operation with the filter size of 7×7 and σ denoted the sigmoid function according to [12]. The spatial attention calculated the feature correlation across the channel domain to find the cardinal features across the entire spatial domain.
Loss Function and Training. Our loss function consisted of content loss and adversarial loss. The content loss function was basically made up of three where α, β, γ represented the hyper-parameters.
Most of the GAN based CS-MRI studies used vanilla GAN objective [13], which applied the Kullback-Leibler (KL) divergence, as the adversarial loss function. However, during the training of the generator, when the generator deviated from the optimal solution, the parameters of the generator might not be updated continuously, which could then lead to complicated training process and model collapse [14]. In this study, we introduced WGAN-GP [15] as an alternative strategy of using Wasserstein distance to displace the KL divergence for solving the potential complications in training the GAN. WGAN-GP also introduced the gradient penalty to better solve the common gradient vanishing problem. We used a loss that was calculated as the following In order to improve the perceptual quality, we also incorporated the content loss with three different combinations of the loss functions: We used normalised MSE (NMSE) as the optimisation cost function. However, the use of NMSE as content loss alone might lead to perceptually uneven reconstruction, resulting in a lack of coherent image details. Therefore, to consider the perceptual similarity of images, we also added NMSE of the frequency domain data and VGG loss (L VGG ) as additional constraints.

Experiments
Datasets. Our experiments were performed on two datasets (1) Brain MRI dataset: We trained and tested our model using a MICCAI 2013 grand challenge dataset. In total, we included 726 3D data for our study. We randomly used 503 data for training, 173 for validation and 50 for testing. Each 2D slice had a shape of 256×256, and we normalised the intensities into a range of [- 1 1]. (2) Cardiac MRI dataset: A population of 100 3D LGE MRI patient data, which were made available through the 2018 Atrial Segmentation Challenge, were used in this work. The scanner used for this clinical study was a whole-body MRI scanner, within an image acquisition resolution of 0.625mm 3 . The studied data were randomly divided into 80 for training, 10 for validation and 10 for testing. Similarly, we normalised each slice into a range of [- 1 1].
Experiments Setup. For all the input data, we applied data augmentation on the input 2D image slices. Besides, we used raw k -space data with different undersampling ratios to simulate the corresponding acceleration factors. In particular, 10%, 30% and 50% retained raw k -space data were simulated representing 10×, 3.3×, 2× accelerations assuming that the preparation time of MRI scanning is insignificant. All our comparison studies were carried out using different CS-MRI reconstruction algorithms using these three levels of undersampling ratios. Our studies were mainly divided into three experiments: First, We compared the performance of our method with that of other SOTA at three different acceleration factors. In addition to the traditional metrics of PSNR, we also introduced the mean opinion scores (MOS) to take human perception into account, which was the results of domain experts evaluating the reconstructions and averaging their perceptual quality. Then, at different acceleration factors, we tested the noise reduction effect of all models at different noise level to prove that our model could significantly suppress the residual noise. To test the noise tolerance of different CS-MRI methods, we added white Gaussian noises to the k -space data before applying the undersampling. Inspired by [16], we conducted a noise level estimation for all the reconstruction results. Finally, we tested the effectiveness of various network configurations of our proposed framework. In the final quantification, we used PSNR, SSIM and NMSE as the evaluation metrics.

Results
Image Quality Comparison. Our method has demonstrated the best performance by comparing with four SATO methods (on both brain and cardiac MRI datasets). Table 1 shows that the results of our proposed DAWGAN performed best in PSNR and MOS. At the 10× and 3.3× acceleration factors, the PSNR and MOS achieved by our DAWGAN were significantly higher than the other methods. In addition, Figure 2 shows that DAWGAN produced less noiseinduced artefacts in all the simulation studies, while the other methods had more noise-induced artefacts. Although CRNN and DAGAN could also suppress some artefacts, the reconstructions of the brain area were less detailed than those  Noise Suppression Comparison. In terms of reconstruction details, we demonstrated that DAWGAN could effectively reduce residual noise in the reconstructed images. As shown in Figure 3, DAWGAN suppressed the noise effectively at different noise levels. Figure 4 shows the PSNR results with respect to different noise levels and various undersampling patterns. Our proposed DAW-GAN also demonstrated considerable noise tolerance at different noise levels, and the mean value of the PSNR was higher than other methods.
Ablation Studies. The performance of our framework with various network components was shown by our ablation studies. The sub-models we compared were WGAN-GP+RNN, WGAN-GP+Attention and Attention+RNN. Table 2 shows that the DAWGAN full model was superior to other sub-model variations using all three metrics, which indicated that the current configurations in our proposed network architecture are effective.

Conclusion
In this paper, we proposed the DAWGAN to reconstruct MRI images from highly undersampled k -space data. Our DAWGAN employed WGAN-GP to improve the stability of vanilla GAN. The incorporated Bi-ConvLSTM block can make full use of the relationships among successive MRI slices to improve the reconstruction results. In addition, the proposed SAB can distinguish between significant and non-significant features for our MRI reconstruction task. Our ablation studies have demonstrated the effectiveness of the key components of our framework. The comprehensive comparison studies on both brain and cardiac MRI datasets have corroborated that our method can not only achieve better reconstruction results but can also effectively reduce residual noise generated during the reconstruction process.