{'fraction_non_alphanumeric': 0.032444418509657465, 'fraction_numerical': 0.014004784968197468, 'mean_word_length': 4.9651931778628615, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptively Re-weighting Multi-Loss Untrained Transformer for Sparse-View Cone-Beam CT Reconstruction

Cone-Beam Computed Tomography (CBCT) has been proven useful in diagnosis, but how to shorten scanning time with lower radiation dosage and how to efficiently reconstruct 3D image remain as the main issues for clinical practice. The recent development of tomographic image reconstruction on sparse-view measurements employs deep neural networks in a supervised way to tackle such issues, whereas the success of model training requires quantity and quality of the given paired measurements/images. We propose a novel untrained Transformer to fit the CBCT inverse solver without training data. It is mainly comprised of an untrained 3D Transformer of billions of network weights and a multi-level loss function with variable weights. Unlike conventional deep neural networks (DNNs), there is no requirement of training steps in our approach. Upon observing the hardship of optimising Transformer, the variable weights within the loss function are designed to automatically update together with the iteration process, ultimately stabilising its optimisation. We evaluate the proposed approach on two publicly available datasets: SPARE and Walnut. The results show a significant performance improvement on image quality metrics with streak artefact reduction in the visualisation. We also provide a clinical report by an experienced radiologist to assess our reconstructed images in a diagnosis point of view. The source code and the optimised models are available from the corresponding author on request at the moment.


Introduction
Cone-beam Computerised Tomography (CBCT) has been widely applied in dental and maxillofacial imaging as gold standards, and extended to lung imaging. Compared to conventional 2D CT method, CBCT utilises a fast 3D imaging method to visualise 3D anatomy. Researches show that its radiation dose under full-view acquisition is effectively higher than plain radiographs [18], and hence, sparse-view CBCT is more attractable owing to lower radiation dose and faster scanning time compared to full-view CBCT. However, the classical reconstruction methods for sparse-view CBCT, for example back projection-based [9] and compressed sensing-based [3] approaches may reconstruct sub-optimal image with streak artefacts and may have slow convergence of iterations.
Broadly, there is a surge on adopting deep neural networks (DNNs) to tomographic imaging reconstruction, where DNNs mostly regularise the raw measurements/images or invert high-quality images from the measurements in a supervised way [22,23]. For example, Jiang et al apply the symmetric residual DNN to CBCT reconstruction [12]. Zhang et al enhance CBCT image quality using the generative adversarial network [26]. The success of model training substantially relies on the quantity and quality of the given paired data, whereas tomographic measurements are rarely available in the clinical scenario. On the other hand, simulation helps generate a large-scale dataset but the model trained on emulated data may have limited generalisation ability. Generative adversarial network-based model extrapolates super sparse-view training data to have better perceptual visualisation of scans, but may suffer from the hallucination that adds false positive structure into the image anatomy [2].
We propose an untrained Transformer having billions of network weights to fit a CBCT inverse solver without large number of data, ground-truth images, or geometric information when the reference images are provided. Fitting the Transformer model adapts the deep image prior (DIP) technology that is used to recurrently optimise network parameters to restore better image in neural representation [6,21]. Using Transformer overcomes the limitation of small capacity networks and further enhances the reference images. We adopt UNEt TRansformer (UNETR) [11] as the exampled Transformer architecture. We further incorporate mean squared error (MSE) and perceptual loss (PL) [13] to obtain both smooth and visually pleasing output of CBCT reconstruction. However, Transformer is notoriously hard to optimise, especially when multi-loss functions are used [4]. We propose an automatic re-weight technique that locally optimises the multi-loss weights along with the iteration of Transformer weights.
Our main contributions are: 1) Optimising the large-scale 3D Transformer with only one reference data but without ground truth data; 2) Stabilising the iterative optimisation of multi-loss untrained Transformer via re-weighting technique; 3) Extending the most DIP research on 2D to 3D medical imaging scenario; 4) Breaking the jail of cross-scanner and cross-protocol issues that commonly barrier the generalisability of the trained model in machine learning-based medical imaging problems. Our overall paradigm is not limited to UNETR, and can extend to other vision Transformers such as Swin Transformer [16] or SEgmentation TRansformers (SETR) [24].

Cone-Beam CT with Untrained Neural Networks
To formulate the 3D CBCT problem, we denote X the imaging region-of-interest (ROI) of N 1 × N 2 × N 3 voxels and Y the M -view measurements acquired on a Fig. 1. The framework of the proposed multi-loss untrained networks for sparse-view CBCT reconstruction. Three main modules in our proposed approach function to reconstruct the referred CBCT image by the FDK algorithm, to generate the Transformer prior-embedded image, and to extract multi-level image features using the VGG-11 network. The referred and the generated images are inputted to the per-pixel meansquared-error (MSE) loss. The multi-level image features are inputted to the weighted multi-objective perceptual loss (PL). Dimension Shrinkage Convolution (DSConv) can learn to reduce 3D features to 2D features of 3 channels. The abbreviations of Conv, ReLU, and FC refer to convolution operation, rectified linear unit, and fully-connected layer. Red fonts indicate the variable weights for optimisation. detector of D 1 × D 2 pixels. We propose to invert the regularised CBCT image X from the given measurements Y by the framework illustrated in Figure 1.

Network Modules
The entire framework is comprised of three modules: -Image Reconstructor: We use the Feldkamp-Davis-Kress (FDK) algorithm [9] to recover the referenceX having same dimensions as X from the given measurements Y . -Image Generator: 16 random noise volumes denoted by ε are fed into untrained deep neural networks denoted by G : R 16×N1×N2×N3 → R N1×N2×N3 , says Transformer architecture, to generate the regularised CBCT image X. Our implementation of Transformer will be describe in Section 2.3. -Feature Extractor: Composed of a Dimension-Shrinkage Convolution (DSConv) and a pre-trained VGG-11 [20] feature extractor. The DSConv operation C : R N1×N2×N3 → R 3×N2×N3 enables the reduction of the first dimension of output image X to 3, subsequently matching the input shape of the VGG-11 network.
As inspired by DIP [21] and Deep Decoder [6], we optimise the network weights of the untrained Transformer requiring only the sparse-view measurement data.
The network weights are considered as a parameterisation of the full-view like CBCT image once loss function is converged.

Multi-Level Perceptual Loss
We incorporate a variant of PL with MSE to serve as the total loss; see Figure 1. Unlike the original PL only making use of the high-level features in pre-trained networks, we utilise a mixture of all the activation maps in those networks, aiming to remove streak artefacts based on all-level perceptual features. Hence, the total loss function can be written as where θ denotes the network weights in the Image Generator module, α the regularisation parameter, w = (w 1 , · · · , w K ) the weights of multi-level PL of K levels, ϕ andφ respectively the weights of the reconstruction DSConv C ϕ and the generation DSConvĈφ, and f i the intermediate output of feature extractor at the ith Rectified Linear Unit (ReLU) layer. We freeze the network weights of the two VGG-11 feature extractors that were pre-trained on ImageNet [7].

Adaptively Re-weight Multi-Loss
In Equation 1, a successful choice of multi-level PL weights w ensures stable optimisation of our proposed method. Instead of using a group of fixed heuristic values, we propose a novel adaptive way that regards w as differentiable variables and subsequently re-weight them along with the network weights. As demonstrated in Algorithm 1, we use automatic differentiation (AD) to compute gradient, and use gradient descend (GD) (or ADAM [15] as an advanced option) to update weights. Moreover, we observed that some adaptive weight component w i may occasionally alter to a negative number, leading to negative loss value and hence substantially degraded reconstruction performance. Through our empirical study, normalising w can stabilise the optimisation step. To tackle this problem, we employ the clip/gradient clip and the subsequent normalisation to w during the iterative process.

Transformer Architecture
We adopt a variant of UNETR [11] as our implemented Transformer architecture for the Image Generator module, without the use of activation layer before outputting images. Basically, UNETR has a U-Net [5] architecture that includes the following modules: token (or 3D patch) extraction, linear projection added

Algorithm 1 Adaptively Reweight Multi-Loss
We use Gradient Descent here but still can extend to ADAM. Compute: X θ t+1 = G θ t+1 (ε). end for with positional embedding, Transformer encoding, DNNs decoding with U-Net style skip connection. The detailed illustration can be found in Figure S4.
In specific, we first input 16 random noise volumes of a uniform size of N 1 × N 2 × N 3 . All input volumes are then divided into uniform non-overlapping 16 × 16 × 16 tokens, totally forming a token sequence of the length n. Each token being flatten and its position code separately project to K dimensional space and then sum up together. Next, we utilise twelve Transformer blocks to perform encoding process. Each block includes a normalisation layer, the multi-head selfattention (MSA), a summation operation, a normalisation layer, the multilayer perceptron (MLP) sublayers and a summation operation in a sequential order. The summation operation adds the input feature and its corresponding output together via skip connections. Lastly, the expansion path of DNNs decoding is the same as the 3D U-Net, whereas we require the output of Transformer block to have equal resolutions with same level decoder features by several deconvolution operations of a upsampling factor of 2 before concatenation. Two consecutive 3 × 3 × 3 convolutions with Batch Normalisation and ReLU subsequently follows the last deconvolution operation. Such process is repeated for all subsequent layers until the final output has the same resolution as the original input.

Data Description
We perform our study on the publicly available the SPArse-view REconstruction (SPARE) Challenge dataset (abbreviated as SPARE) [19] and Walnut dataset [8]. As for the SPARE dataset, five patients are scanned on an Elekta Synergy scanner (Elekta, Stockholm, Sweden). A cone-beam source emits approximately 1008-1022 projection views uniformly on 180 degree using the full-fan mode separately onto a 0.8 mm-resolution detector of 512 × 512 pixels. The region-of-interest is set on lung with 176 × 176 × 176 voxels with the size 1 mm. The source-toobject distance (SOD) and the source-to-detector distance (SDD) are 1000 mm and 1536 mm, respectively. The FDK algorithm is performed by the ASTRA Table 1. Evaluation results for the reconstruction models of FDK, SIRT, and DIPs using 3D U-Net, UNETR and our approach. All models were performed on the SPARE and the Walnut datasets. The mean and the standard deviation (std) over all subjects in the datasets were computed. Bold font denotes the best mean. The asterisk * denotes p-value< 0.05 compared with the rest methods in the sense of the one-sided Wilcoxon signed-rank test.

Implementation Details
The proposed method was implemented on Pytorch 1.9 [17] and was run on a single Nvidia Tesla V100 GPU with 32 gigabytes memory. In terms of classical reconstruction algorithms, we used FDK and Simultaneous Iterative Reconstruction Technique (SIRT) [14] algorithms in ASTRA-toolbox 2.0 to obtain sparseview reconstructed images. The iteration number in SIRT was set to 200. The ADAM optimiser was used with a starting learning rate of 10 −3 and the decay was 10 −5 . The regularisation parameter α was 1. The inputs of our proposed network were 16 random noise volumes drawn from the 3D standard Gaussian distribution with the size same as the reference image. We initialised all network weights with Glorot normal initialiser [10]. For the UNETR backbone, the patch size (or the token size) was set to 16 × 16 × 16, the multi-heads number was 12 with 768 embedding size, and the MLP output size was 3072. To unify input feature dimensions, the patch-based 3D U-Net baseline chose the same input shape of 16 × 16 × 16. All DNN-based methods terminated at 2000th iteration.

Results
The comparative study was carried out on FDK, SIRT, 3D U-Net [5], UNETR and our proposed method. The DIPs was used in the latter three approaches. All the neural networks had comparable model capacity. We employed a onesided Wilcoxon signed-rank test to determine the statistical significance of the performance difference between two comparing methods. Table 1 showed that for both SPARE and Walnut datasets, our approach significantly (p-value < 0.05) outperformed all other methods measured by peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [25]. To visualise reconstruction performance in an individual volume map, Figure 2 compared the sagittal planes of different reconstructed results using a single patient measurement of 100-view points from the SPARE dataset. We observed that our proposed approach enhanced the resolution both in a zoomed region and showed lower error score in the image domain compared to the other baseline methods. The qualitative analysis was done by an experienced radiologist who was specialised in lung CT image analysis. The report indicated that the GT image and our result apparently visualised anatomical structure of the main bronchus. The images reconstructed by FDK, U-Net and UNETR showed no clear vision of bronchus due to background noise and low contrast issue. The reconstructed image by SIRT demonstrated severe distortion and spike artefact.
We further conducted an ablation study of three different weighting strategies for our proposed approach. They were switching-off the multi-level perceptual loss (denoted by w = 0), fixing weights for the multi-level perceptual loss (w = 1) and automatically reweighting the multi-level perceptual loss (Reweight w). In Figure 3(a), we observed that for all the weighting strategies, the iteration curves in terms of the PSNR score between the iteration output and the FDK reference globally increased despite the several "dips" (also observed in [4]), which might partially fail the optimisation of our approach. The case w = 0 strategy had higher performance than the other two, however, this had an in- coherent trend between the iteration output and the GT image, as illustrated in Figure 3(b). The cases w = 1 and the case Reweighting w showed consistent performance increase in terms of both the FDK reference and the GT image, but the case Reweighting w had fewer "dips" hence more stable than the case w = 1. Additional experiments on the robustness of our method on different numbers of projection views and the visualisation results for the Walnut dataset can be found in the Figures S5 and S6, respectively.

Discussion and Conclusion
Our proposed approach is a novel multi-loss DIP approach on an untrained Transformer for the sparse-view CBCT reconstruction. We extend the conventional DIP to 3D large-scale network appended with multi-level PL, and subsequently overcome the difficulty of both optimising the used Transformer and reweighting multi-loss. We quantitatively and qualitatively demonstrate that the proposed approach enhances the reconstructed image quality and stabilises the convergence to the GT image, coherently uniform with the observable trends to the FDK reference. Our proposed Transformer-based multi-loss DIP offers several avenues for future improvement and application. First, optimisation of the multi-level PL highly depends on a proper choice of a pre-trained network (e.g. VGG-11); other feature extractors for medical images such as Pre-trained Models Genesis [27] may be worth investigating. Broadening generalisation bound of the DIP-type method may be another working direction [28]. On the other hand, our approach can potentially extend to other tomographic imaging modalities, resulting in speeding up the acquisition, reducing radiation exposure, and increasing the patient throughput. In future work, we will evaluate the clinical viability of our approach for CBCT imaging.