{'fraction_non_alphanumeric': 0.030975551863280323, 'fraction_numerical': 0.016536118363794605, 'mean_word_length': 4.825996773450104, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep learning-based real-time volumetric imaging for lung stereotactic body radiation therapy: a proof of concept study.

Due to the inter- and intra- variation of respiratory motion, it is highly desired to provide real-time volumetric images during the treatment delivery of lung stereotactic body radiation therapy (SBRT) for accurate and active motion management. In this proof-of-concept study, we propose a novel generative adversarial network integrated with perceptual supervision to derive instantaneous volumetric images from a single 2D projection. Our proposed network, named TransNet, consists of three modules, i.e., encoding, transformation and decoding modules. Rather than only using image distance loss between the generated 3D images and the ground truth 3D CT images to supervise the network, perceptual loss in feature space is integrated into loss function to force the TransNet to yield accurate lung boundary. Adversarial supervision is also used to improve the realism of generated 3D images. We conducted a simulation study on 20 patient cases, who had received lung SBRT treatments in our institution and undergone 4D-CT simulation, and evaluated the efficacy and consistency of our method for four different projection angles, i.e., 0°, 30°, 60° and 90°. For each 3D CT image set of a breathing phase, we simulated its 2D projections at these angles. For each projection angle, a patient's 3D CT images of 9 phases and the corresponding 2D projection data were used to train our network for that specific patient, with the remaining phase used for testing. The MAE of the 3D images obtained by our method are 99.3±14.1 HU. The PSNR and structural similarity index metric (SSIM) within the tumor region of interest (ROI) are 15.4±2.5 dB and 0.839±0.090, respectively. These results demonstrate the feasibility and efficacy of our 2D-to-3D method for lung cancer patients, which provides a potential solution for in-treatment real-time on-board volumetric imaging for accurate dose delivery to ensure the effectiveness of lung SBRT treatment.


INTRODUCTION
Respiratory motion causes substantial anatomic changes, leading to significant dosimetric uncertainties in lung cancer radiotherapy [8,9]. The impact of these uncertainties is amplified in lung stereotactic body radiation therapy (SBRT), which delivers very high and conformal doses in relatively few fractions [3,18]. Motion management is hence essential for treatment accuracy and planning margin reduction in lung SBRT [1,8]. Over the years, time-resolved imaging modalities have been developed and adopted clinically for motion management, such as 4DCT which is routinely used for treatment planning [9,16,8], and 4D cone beam CT which is an emerging imaging tool to aid patient positioning [12,17,20]. Nevertheless, respiratory motion can vary between CT simulation, patient positioning and treatment delivery in its magnitude, baseline, period and regularity [28]. Hence, in-treatment real-time imaging is highly desired to provide the actual anatomy information during treatment delivery for in-treatment motion monitoring and management [9,27].
Projection based 2D imaging, such as kV fluoroscopy and MV Beam Eye View images is a common way to realize in-treatment real-time imaging [22]. However, because the patients 3D anatomy is projected and superimposed onto a 2D plane, the visibility of the tumor is very limited, especially when it is behind a high intensity structure such as bone [27]. The on-board imaging component of MRIdian MRI-guided linear accelerator (ViewRay company, Cleveland, Ohio) is a pioneering system that can perform real-time imaging in a single 2D pane without the aforementioned superimposing issue [25]. However, the unobserved tumor/OAR motion in other planes during treatment delivery might impair the tracking accuracy [6], let alone the high cost of this system and the interference between its magnetic field and the radiation beams. Another real-time 2D imaging scheme has also been proposed to capture the instantaneous anatomy on a 2D plane by measuring the scatters of the therapeutic beams during treatment, which demands a super sensitive detector to capture the relatively small number of scattered photons and has the same issue of unobserved motion on other plans as cine-MRI [27].
Because of the wide availability of on-board imaging system integrated in Linac machines, it is highly desired to generate instantaneous volumetric image using this on-board imager. Due to the slow rotation of Linac gantry, it is impossible to acquire sufficient angular projection data at high frequency to satisfy the Shannon-Nyquist sampling theorem for traditional computed tomography or tomosynthesis approaches to derive the instantaneous volumetric image [24]. Although compressed sensing techniques have been employed to successfully reduce the required number of projections to achieve low dose CBCT [7,19], the resultant sparsity is far away from meeting the demand of real-time volumetric imaging [24]. Recently, efforts have been made to derive a volumetric image from a single x-ray projection either using 2D-3D registration techniques [11,13] or machine learning techniques [26,29], in combination with a lung motion model as a prior knowledge. However, the robustness of these techniques to handle the variation in respiratory motion, patient positioning and anatomy change is still a concern for clinical application [27].
Deep learning techniques have attracted much attention for their ability to learn complex relationships and to incorporate existing knowledge into the inference model [10,23], and have found widespread applications across different disciplines [2,4,14,15,21]. Shen et al. has recently trained a deep neural network to learn the feature-space transformation between a 2D projection and a 3D volumetric CT image, and used this trained model to derive volumetric CT image from an anterior-posterior single projection view [24]. Inspired by this novel idea, we propose to employ the advanced perceptually supervised Generative Adversarial Networks (GAN)-based approach to generate volumetric image from a single projection and have tested its effectiveness and robustness for different projection angles. This approach has 2 distinctive strengths: 1) By adding adversarial loss to differentiate the distribution of generated 3D volume from ground truth 3D volume, the capability of our method to generate more realistic 3D CT is significantly improved. 2) By adding perceptual loss in feature space, our networks are enforced to yield more accurate contrast around lung organ. This proof-of-concept study is expected to pave the way for real-time volumetric imaging for more accurate and active motion management during treatment delivery of lung SBRT treatment.

System overview and network design
Our implementation of the transformation from 2D projection to 3D volumetric image consists of two major stages: a training stage and a transform stage. Fig. 1 shows a schematic flow chart of the training stage. At this stage, a feed forward path, named TransNet, is trained under several supervision mechanisms that perform learnable parameters optimization to learn the 2D-to-3D mapping. A set of paired 2D projection (with angle n and size of h 1 × w 1 ) and 3D CT volume (with size of h 2 × w 2 × d) is served as the training data set. TransNet takes a single 2D projection as input, and outputs a generated synthetic 3D CT. The corresponding real 3D CT is used as ground truth to supervise TransNet.
TransNet consists of three subnetworks, i.e., an encoding module, a transformation module and a decoding module. The encoding module is to explore 2D features from the 2D projection data. It is composed of several 2D residual blocks [5] and three 2D max pooling layers to reduce the feature map size to h 1 Here, n f denotes the number of feature maps extracted from the first residual block of encoding module. After this module, the number of feature maps is increased to 8 • n f . The following transformation module is to bridge the feature maps from 2D scene to 3D scene. It uses tensor reshaping to transform the 2D feature map to the corresponding 3D one), and employs several 3D convolution layers with valid padding and a tensor bicubic interpolation to resize the transformed 3D feature map to a size of h 2 The decoding module then generates transformed 3D CT image (with a size of h 2 × w 2 × d) from these 3D feature maps. This module is composed of several 3D deconvolution layers, each followed with a 3D convolution layer and an extra 3D convolution layer to perform regression.
The transform stage follows the same feed forward path in the training stage, that is, extracting 2D feature from the input 2D projection and feeding it into TransNet to derive the 3D CT volume.

Supervision and loss function
The performance of this proposed TransNet heavily relies on the loss function used for supervision, which is formulated as: in our study. Here, L mae (ŷ, y) denotes the mean absolute error (MAE) between transformed 3D CTŷ and ground truth 3D CT y to measure the voxel-level error. L gd (ŷ, y) denotes the gradient difference (GD) to measure the gradient-level error. These two terms are used as image distance loss between the transformed 3D CT and the corresponding ground truth 3D CT. L p fŷ, f y denotes the perceptual loss, which is defined in feature domain instead of image domain and calculates the difference between the feature map of the transformed 3D CT (i.e., fŷ) and the feature map of the ground truth (i.e., f y ). The last term, L adv (ŷ), denotes the adversarial loss, which is deployed to force TransNet to yield realistic 3D images. This perceptual loss and adversarial loss will be present in details in subsection 2.3 and 2.4, respectively. λ p , λ mae and λ gd denote the balancing hyper-parameters of perceptual, MAE and GD loss, respectively. TransNet is trained via minimizing this total loss.

Perceptual loss
Mapping a single 2D projection to the much more informative 3D image is a very ill-posed problem.
If we only use the image distance loss (e.g., MAE and GD), it is very challenging for TransNet to well represent the correlation between the image difference in 2D projection induced by respiratory motion and the corresponding image difference in 3D volume, resulting in inaccurate or blurry boundary of the moving organs. Since lung is one of the organs that are most susceptible to respiratory motion in thoracic CT images, and lung toxicity is one of the major concerns for lung SBRT treatment, we therefore proposed to integrate perceptual supervision into our networks in our study to enforce more power on generating accurate lung boundary in the transformed 3D images.
The main idea of perceptual supervision [8] is that feed forward networks (i.e., TransNet in this work) could generate high-confidence fooling image (i.e., transformed 3D CT in this work) by using a perceptual loss that measures the perceptual and semantic difference between transformed 3D volume and ground truth 3D volume. In our study, we define the perceptual loss as the feature difference on highlevel feature maps. These high-level feature maps are extracted from both transformed 3D volume and ground truth 3D volume, via a network named SegNet pretrained for image semantic segmentation. As shown in Fig.1, SegNet uses the traditional end-to-end feature pyramid network (FPN) architecture, and extracts pyramid feature maps from both transformed and ground truth 3D volumes for perceptual supervision. It was pre-trained using the dataset of chest CT images and paired lung contours obtained from 2017 AAPM Thoracic Auto-segmentation Challenge [3,1].
Denoting TransNet as F e • F t • F d , the transformation of an input 2D projection x to a 3D volumeŷ can be denoted byŷ = F d (F t (F e (x))). SegNet, denoted by F s , extracts multi-level feature maps from the 3D ground truth CT (y) and the transformed one (ŷ), respectively, i.e., where N is the number of pyramid levels. The perceptual loss is defined as the Euclidean distance between the two feature maps, calculated as where C i denotes the number of feature map channels at ith pyramid level. H i , W i and D i denotes the height, width and depth of that feature map. According to Fig. 1, these values are calculated as . ω i is a balancing parameter for feature level i. Since the semantic information of the feature map at higher pyramid levels would be coarse, the weight for that levels perceptual loss should be enlarged, thus we chose ω i = 1.2 i−1 .

Adversarial loss
Apart from accurate lung boundary, another requirement on the derived synthetic 3D images is to be realistic compared to the ground truth images. Due to the motion artifact, the ill-posed TransNet would introduce additional noise or bias into the transformed 3D CT. Our aim is to reduce this artificial noise via enhancing the realism of transformed 3D CT.
In previous conditional generative adversarial networks (CGAN) studies [5,18], the common way to improve the outputs realism is to use a discriminator network to discriminate the synthetic one (i.e., transformed 3D CT in this task) from real one (i.e., ground truth 3D CT in this task). The discriminator used in our study is a traditional fully convolution network (FCN). It is composed of several 3D convolution layers and 3D max-pooling layers to reduce the size of feature map to 1111, and a sigmoid layer to binarize the output (i.e., 1 means that the discriminator think the input is real, and 0 means synthetic). Denoting the discriminator as F dis , the discriminator loss is to measure its recognition error, calculated as Here, F dis (•) is a binary value indicating whether the discriminator determines the input to be real or synthetic image. The function SCE (•, •) is the sigmoid cross entropy between two distributions. The more similar the two distributions are, the smaller the value of sigmoid cross entropy is. The adversarial loss is to measure the recognition accuracy for the 3D images generated by TransNet, calculated as By minimizing both discriminators loss (i.e., minimizing the recognition error of differentiating synthetic images from real images) and generators adversarial loss (i.e., maximizing the recognition error for high-quality 3D image that can fool the discriminator), TransNet can yield a 3D volume of a comparable realistic level with the ground truth 3D volume.

Training and testing
Our method was implemented in python 3.6.8 and TensorFlow 1.8.0 on a NVIDIA Tesla V100 GPU with 32GB of memory. Adam gradient optimizer with learning rate of 2e-4 was used for optimization. Currently, we do not have in-treatment instantaneous volumetric image technically available for us to use as ground truth. Hence, in this proof-of-concept study, we use 4D CT images of our previous lung cancer patients, which were acquired for treatment planning, to test the feasibility of the proposed perceptually supervised GAN-based volumetric imaging method. 4D CT images of 20 previous patients were collected and anonymized. Institutional review broad approval was obtained; informed consent was not required for this Health Insurance Portability and Accountability Act (HIPPA)-complaint retrospective analysis. All these 4D CT images were acquired on a Siemens SOMATOM Definition AS CT scanner with a resolution of 0.9770.9772 mm 3 . Each 4D CT image consists of 10 3D CT sets, corresponding to 10 respiratory phases.
In our study, apart from the anterior-posterior projection angle (i.e., 0), we have also tested three other projection angles 30, 60 or 90, respectively. For each 3D CT image set, 2D projection data at these angles were generated by forward-projecting the CT volume to a plane using the same geometry as an on-board imager of Varian (Palo Alto, CA) Truebeam Linac. 40 experiments were performed for each patient, varying the combination of phase and projection angle. Specifically, at each experiment, our networks were trained and tested for one of the four projection angles (0, 30, 60 or 90) in one phase. The pairs of 2D projection and the corresponding 3D CT image of 9 breathing phases were used as the training data, and the remaining phase was used for testing. We repeated this experiment 40 times to let each angle with each phase used as test data exactly once. To further enlarge the variance of training data, data augmentation, which includes rotation, flipping and scaling, was applied.
Four metrics were used in our study for quantitative comparison, that is, mean absolute error (MAE), normalized MAE (NMAE), peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM). MAE is a magnitude of the absolute difference between the transformed 3D CT and ground truth 3D CT. NMAE is a magnitude of the absolute difference between the normalized transformed 3D CT and normalized ground truth 3D CT (normalized images to [0, 1] using a same normalization factor). PSNR is used to compare the noise level between the transformed 3D CT and ground truth 3D CT. SSIM, which is commonly used in pattern matching and image analysis, is a measure of the similarity of image structures. The calculations of these metrics were conducted over the entire image volume.
To demonstrate the contribution of perceptual supervision and adversarial supervision to the final performance, a step-by-step study was performed to evaluate the effectiveness of each component. In order to illustrate the statistical significance of quantitative improvement by these steps, paired twotailed t-tests were also performed among 10 phases for each patient.

Efficacy of perceptual supervision
To demonstrate the efficacy of the perceptual supervision, we compared the transformation accuracy of the proposed TransNet between two different supervision strategies, i.e., 1) being supervised by MAE and GD losses (referred to as TransNet-v1), and 2) being supervised by a combination of MAE, GD and perceptual losses (referred to as TransNet-v2). This comparison was conducted on all the patients at one projection angle (60) for demonstration purpose. It can be observed from these images that the feature map of TransNet-v2 around lung boundary is more similar to that of ground truth, which leads to less blurry and more accurate lung boundary in TransNet-v2 than TransNet-v1. From the line profile of yellow line on ground truth CT, TransNet-v1 and TransNet-v2 generated 3D CTs (shown in e2), we can see that the generated 3D volume via TransNet-v2 has a more accurate lung contrast as compared to TransNet-v1.
The quantitative evaluation results of the efficacy of perceptual supervision are listed in Table 1

Efficacy of adversarial supervision
To demonstrate the efficacy of the adversarial supervision, we compared the transformation accuracy between TransNet-v2, which is supervised by a combination of MAE, GD and perceptual losses, and TransNet-v3, which is supervised by a combination of MAE, GD, perceptual and adversarial losses. This comparison was conducted on all the patients at projection angle = 90 (left-right projection view) for demonstration purpose. Fig. 3 shows the visual comparison between TransNet-v2 and TransNet-v3 for a patient. An axial slice of the ground truth 3D CT, and the 3D CT generated by TransNet-v2 and TransNet-v3 are shown in subfigure (a1) -(a3), respectively, and the corresponding zoomed-in images are shown in (b1) -(b3). As can be seen from these images, TranNet-v2 seems to induce noise on the generated 3D images, particularly around the chest wall-lung interfaces. By using adversarial supervision, TransNet-v3 can effectively remove these artificial noises imposed by TranNet-v2. The performance of TransNet-v3 versus TransNet-v2 can be also seen from the line profile in (c2). To quantitatively evaluate the efficacy of the adversarial supervision, MAE, NMAE, PSNR and SSIM were calculated for the transformed 3D CT generated by TransNet-v2 and TransNet-v3, respectively, and shown in Table 2 Table 3-6, respectively. These numerical results have well demonstrated the robustness of our proposed method at different projection angles. It can be also observed that among these four angles, 0 projection angle (i.e., anterior-posterior projection view) yielded the best performance, while 30 projection angle (i.e., left anterior-oblique projection view) yielded the relatively worst performance.

Discussion
Shen et al. proposed to employ a deep neural network, for the first time, to learn the mapping relationship between a single 2D projection radiograph of a patient to the corresponding 3D anatomy, and used this trained network to generate volumetric tomographic X-ray images of the patient from a single anterior-posterior projection view [24].
In their experiments, they tested on one patients lung 4D CT images and achieved 0.025 NMAE, 27.157 dB PSNR, and 0.838 SSIM. Inspired by this work, we have employed a more advanced GAN network for this ill-posted 2D-to-3D transformation problem, and integrated perceptual supervision into it. As demonstrated in Fig.2 and Table 1, adding the perceptual supervision to our network can significantly enhance its sensitivity for accurate lung boundary regression. In addition, the adversarial supervision significantly improved the realism of the generated 3D images in relative to the ground truth images, as demonstrated in Fig.3 and Table 2. The efficacy and consistency of our method at different projection angles have also been well demonstrated by our experimental results in Fig. 4 and Table 3-6. At the anterior-posterior projection angle, the NMAE value achieved by our method for the 20 patient cases ranges from 0.019 to 0.044, with a mean value of 0.029 and a standard deviation of 0.008. The achieved PSNR value ranges from 20.1 to 28.4 dB, with a mean value of 24.1 dB and a standard deviation of 2.4 dB. The achieved SSIM value ranges from 0.907 to 0.973, with a mean value of 0.956 and a standard deviation of 0.012. Although it is difficult to fairly compare our method with Shens method since 20 different patient cases were used in our study, we can observe that our method achieves comparable performance for the anterior-posterior projection angle. In our study, we have also found that the anterior-posterior projection angle (i.e., 0) yielded the best performance among the four testing projection angles. This might be due to the thinnest patient thickness at this angle, which results in more pixels of useful information with less superimposition on each pixel and hence leads to a less ill-posed problem.
In this preliminary study, we pre-trained the SegNet segmentation network only for lung to see whether the perceptual supervision can help improve the accuracy of lung boundary on the generated 3D images. With the obvious efficacy of this perceptual supervision being observed from our experimental results, in future we will extend our perceptual supervision to other moving organs, such as heart and esophagus, and tumor for more accurate image boundary of these moving objects.
This proof-of-concept study has demonstrated the feasibility of deriving a 3D volumetric image from a single 2D projection of different projection angle, paving the way for in-treatment real-time volumetric imaging using the on-board imager. However, there are still two challenges that need to be overcome. First, without any projection images and corresponding 3D images available before the actual treatment, we have to use the patients 4D planning CT images (i.e., 3D CT images at each phase) and simulated projection data to train the patient-specific networks. These simulated projections for training need to have similar image quality compared to the actual in-treatment 2D kV projections that will be used to drive the instantaneous 3D images. Conventional ray-tracing methods usually yield ideal projections, while actual kV projections acquired by flat panel detector are contaminated by scatters and noises. This disparity can be addressed by Monte Carlo simulation, which can simulate all possible physics events for each imaging photon to yield realistic projection data for training. Another challenge is the limited size of the on-board imager, which may not cover the entire contralateral lung for patients of relatively large body size, leading to a limited field of view (FOV) in the generated 3D image. This limited FOV might be extended using the 4D CT images and a lung motion model as a prior knowledge. We will focus our future work on the investigation of the potential solutions to address these two issues.

Conclusion
In summary, we have proposed a novel perceptual supervision-based GAN method to derive instantaneous volumetric image from a single projection. The efficacy and robustness of our method have been tested for different projection angles in this study. By validating the feasibility of deriving more informative 3D image from a single 2D projection image at different projection angle via our proposed method, this proof-of-concept study paves the way for in-treatment on-board real-time volumetric imaging for more accurate and active motion management during lung SBRT treatment.