{'fraction_non_alphanumeric': 0.034859021735967645, 'fraction_numerical': 0.017400413187068992, 'mean_word_length': 4.830025445292621, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lung2Lung: Volumetric Style Transfer with Self-Ensembling for High-Resolution Cross-Volume Computed Tomography

Chest computed tomography (CT) at inspiration is often complemented by an expiratory CT for identifying peripheral airways disease in the form of air trapping. Additionally, co-registered inspiratory-expiratory volumes are used to derive several clinically relevant measures of local lung function. Acquiring CT at different volumes, however, increases radiation dosage, acquisition time, and may not be achievable due to various complications, limiting the utility of registration-based measures, To address this, we propose Lung2Lung - a style-based generative adversarial approach for translating CT images from end-inspiratory to end-expiratory volume. Lung2Lung addresses several limitations of the traditional generative models including slicewise discontinuities, limited size of generated volumes, and their inability to model neural style at a volumetric level. We introduce multiview perceptual similarity (MEAL) to capture neural styles in 3D. To incorporate global information into the training process and refine the output of our model, we also propose self-ensembling (SE). Lung2Lung, with MEAL and SE, is able to generate large 3D volumes of size 320 x 320 x 320 that are validated using a diverse cohort of 1500 subjects with varying disease severity. The model shows superior performance against several state-of-the-art 2D and 3D generative models with a peak-signal-to-noise ratio of 24.53 dB and structural similarity of 0.904. Clinical validation shows that the synthetic volumes can be used to reliably extract several clinical endpoints of chronic obstructive pulmonary disease.

obstructive pulmonary disease (COPD) [2]- [6]. For instance, parametric response mapping (PRM) and disease probability maps (DPM) have been used to identify local patterns of emphysema and functional small airways disease (fSAD) from co-registered, inspiratory-expiratory chest CT volumes [3], [4]. PRM and DPM have since gathered widespread clinical attention as they have been used to detect and characterize COPD and asthma [7]. Bodduluri et al. showed that registrationderived local tissue expansion of the lung was associated with respiratory function, emphysema, and patient quality of life [6]. While multiple volumes improve clinical assessment, acquiring them takes more time, increases radiation dose and overall cost, and is not routinely recommended in clinical settings. Moreover, advanced-stage COPD or cancer patients may not be able to reach full inspiration or expiration volumes. These problems limit the utility of multivolume approaches for analyzing lung disease. Cross-volume image synthesis could serve as a useful tool towards mitigating this challenge.
Recently, generative modeling has been used for various image-to-image translation tasks [9], [10]. The most prominent within this paradigm are the generative adversarial networks (GANs) [10], which have been shown to perform remarkably well for various medical image processing tasks such as denoising [11], [12], super-resolution [13]- [15], data augmentation [16], [17], and image-to-image translation [18]- [26]. The GAN framework, which typically consists of two convolutional neural networks (CNNs) competing against each other, can be differentiated by the number of spatial dimensions it attempts to model, i.e., planar (2D) vs. volumetric (3D) approaches. Most of the preliminary work done towards medical image-to-image translation was performed by 2D GAN frameworks that operated on 2D slices instead of 3D volumes. For instance, Yang et al. proposed to jointly minimize adversarial and perceptual loss functions for denoising low-dose CT slices from paired high-dose samples [11]. An image-conditional GAN was trained in a supervised fashion to convert T 1 -weighted MR slices to synthetic CT slices [25]. Zhou et al. developed a feature fusion approach for crossmodal MR image synthesis [23]. They trained three different network backbones using paired 2D slices to fuse representations from different modalities. The multiple-backbone architecture demanded larger memory and the framework was hence constrained to 2D slices [23]. Lyu et al. proposed an ensemble learning approach for MR super-resolution. They improved model performance by using upsampled image priors for training their GAN [15]. Wolterlink et al. used a CycleGAN [27] for unpaired MR to CT slice translation. Overview of the Lung2Lung framework. Lung segmentations of CT volumes at inspiration I T LC and expiration I RV were computed using Seg3DNet [8], followed by co-registration using a DIR method. A generator G and discriminator D were then trained on 3D patches xp and yp, of size p × p × p, from TLC and RV volumes, respectively. Notice, in addition to the adversarial loss L ADV , gradients from the multiview perceptual similarity module ∇L M EAL and multiresolution voxel consistency loss ∇L M R were also back-propagated for learning the generator G. 'M 1 . . . Mn' indicate 3D convolutions with unit filter size (1 × 1 × 1) used in the multiresolution voxel consistency module.
This modeling framework relaxed the assumption of requiring paired training data and utilized two generative models to ensure cycle-consistency between domains [18]. The lack of supervised training in unpaired approaches often leads to structural inconsistencies between real and synthetic images.
Recently, Yang et al. tried to address this by exploiting common structural attributes between two modalities [21]. All of these approaches generate 2D slices which appear discontinuous when stacked together into a volume. Moreover, the slice-based methods are unable to explicitly learn largescale 3D texture patterns which strictly reduces the clinical utility of the synthetic images. The unpaired approaches that rely on multiple generative models incur large GPU memory for training and are increasingly difficult to scale at volumetric levels.
Volumetric (3D) generative frameworks address several limitations presented by the slice-based (2D) methods [16], [19], [20], [22], [26]. Nie et al. trained a 3D GAN on MR image patches to generate corresponding CT patches. They further used auto-context [28] to iteratively refine CT patch synthesis by conditioning each successive GAN on outputs from the previous model [19]. The networks operated on small image patches (of size 32 × 32 × 32) and the overall volume size generated using the trained model was limited to 153 × 193 × 50. The framework additionally proposed to minimize the gradient difference loss to encourage structural consistency between real and synthetic samples [19]. Unlike results from 2D GANs, the synthetic CT volumes were consistent across all three spatial dimensions [19]. Shin et al. also proposed a 3D GAN for data augmentation but had to downsample their image volumes to 128 × 128 × 54 due to limited GPU memory available for training [16]. Recently, another patch-based 3D approach, the edge-aware GANs, leveraged edge detection to better capture structural features for multimodality MR image synthesis [20]. The trained model was demonstrated to generate an overall volume of size 240 × 240 × 155. A multi-resolution cascade of GANs was proposed to alleviate the memory requirements for synthesizing highresolution 3D volumes [26]. Similar to Nie et al., this approach used contextual information from a lower resolution model to train a cascade of patch-based higher resolution models.
Volumetric generative models are typically trained either on smaller image patches or downsampled volumes due to limited GPU memory. Training GANs on patches yields sub-optimal synthesis results due to lack of global context and limited neighborhoods processed by the fully convolutional image generators. This is why most of the aforementioned approaches refined their results either by cascading or additionally regularizing their models with structural constraints [19], [20], [26]. While most of these methods focused on structural constraints, they did not explicitly model the neural style and texture transfer (NST) across modalities. For our task of transforming one CT volume into another, NST offers the potential to model subtle textural details of the lung tissue at different volumes. NST approaches mainly rely on perceptual similarity metrics computed on projections obtained from large, pretrained object-detection models [29]- [31]. A lack of pretrained 3D models that could be used to obtain these highdimensional projections limits the use of NST at a volumetric level. Also, most of the NST approaches require exemplars for stylization that are not available for medical image-to-image translation. Another important limitation presented by most of the GAN studies is a lack of model validation against clinical end-points; a component which was completely missing across all the studies discussed above. Having said that, volumetric GAN-based approaches still offer large, untapped potential for cross-volume CT image synthesis.
In this study, we hypothesized that image-conditional GANs can be used to translate a CT image from end-inspiratory volume to end-expiratory volume. This led us to investigate the role of generative models for cross volume CT synthesis, concurrently addressing the challenges presented above. We propose Lung2Lung -a self-ensembling, style transfer framework -for translating high-resolution CT volumes from inspiration to expiration. The proposed framework consists of a multiview perceptual image similarity module that has been developed specifically for volumetric NST. We show that the Lung2Lung framework, with multiview similarity and self-ensembling, is able to capture subtle tissue textures and generate high-resolution CT volumes up to a size of 320 × 320 × 320 -significantly larger than the volumes generated by most of the 3D GANs mentioned above. The major contributions of our work can be summarized as follows.
• We believe this is the first study that investigates the role of generative models for cross-volume computed tomography, where we present a fully convolutional 3D GAN framework that addresses several limitations of the existing GAN frameworks. • We introduce a multiview perceptual similarity module, that can be seamlessly integrated into the GAN training framework, to better model texture synthesis and style transfer across CT scans at different volumes. • We propose a self-ensembling strategy to better capture global context for volumetric image synthesis, and compare the performance of our method with various state-ofthe-art 2D and 3D image-to-image translation methods. • Unlike most other generative modeling studies, we demonstrate the effectiveness of our model using a large multi-center cohort of almost 1500 participants with varying degrees of disease severity. In addition to routine quantitative evaluation metrics, we demonstrate that synthetic images, derived using Lung2Lung, can be used to reliably compute several clinical end-points such as PRM and air trapping.

II. BACKGROUND
Generative adversarial networks (or GANs) implicitly model the data distribution by learning a mapping G : Z → X from a random noise vector z ∈ Z to a given image x ∈ X , sampled from the data distribution [10]. The GAN framework consists of two different networks that are trained to compete against each other until they reach Nash's equilibrium [10]. This game-theoretic training approach has been successfully used to generate perceptually realistic images for various tasks. Isola et al. extended the original GAN framework to the image conditional GAN (cGAN) that is able to learn a generator G : X → Y, from an image x (in domain X ) to another image y (in domain Y) [32]. The synthetic samplesŷ = G(x), generated by G are then evaluated by a second model called the discriminator D, which is simultaneously trained to learn if the samples are real or fake. Typically, D is a binary classifier that outputs the probability of a sample being derived from the target (or data) distribution. While D is optimized to maximize this probability, the generator G tries to minimize it, which creates a dynamic modeled by the minimax game stated as: where G is trained to minimize (1) such that {G } = min G max D L cGAN . Isola et al. suggested to additionally minimize the pixelwise distance ( 1 or 2 ) between the real y and generated samplesŷ [32]. The minimax objective in (1) can be extended to express the overall generator G loss function as: where the first term in (2) refers to binary cross-entropy (BCE) loss minimized by G, and the second term constitutes the 1distance between the real y and generated samplesŷ. The aforementioned framework is called Pix2Pix and has been shown to be very effective for various 2D image-to-image translation tasks [32].

III. LUNG2LUNG
We introduce a self-ensembling, neural style transfer framework for synthesizing high-resolution 3D CT volumes at endexpiration from the corresponding end-inspiratory volume, shown in Fig. 1. The training framework is preceded by lung segmentation and registration of the inspiratory CT acquired at total lung capacity (TLC) and expiratory CT acquired at residual volume (RV), shown in Fig. 1. We used a tissue intensity and vessel and fissure structure constrained DIR method to match moving RV image I RV to the fixed TLC image I T LC [33]. Lung segmentations of pre-registered RV and TLC volumes were obtained using a cascaded multiscale CNN framework that uses the Seg3DNet architecture [8]. We trained a volumetric image conditional least squares GAN model on image patches (x p and y p ) extracted from I T LC and I RV , respectively. In addition to the adversarial loss L ADV from the discriminator model D, we propose a multiview perceptual similarity (MEAL) loss L M EAL to encourage better style transfer and texture synthesis across volumes. A multiresolution voxel consistency loss (L M R ) is also proposed to encourage high-resolution image synthesis. We refer to the proposed volumetric image-to-image translation framework as Lung2Lung, described in Fig. 1.

A. Conditional Least Squares GAN
We formulate the cross-volume synthesis as an image-toimage translation of a patch x ∈ R 1×H×W ×D from domain of TLC volumes to a patch y ∈ R 1×H×W ×D within the domain of RV scans. The cross entropy-based adversarial feedback in (1) is known to be responsible for vanishing gradients that result in highly unstable GAN training. We used the least squares GAN (LSGAN) framework to stabilize GAN training [34].
, we trained our framework using the following LSGAN objective: To further encourage voxelwise consistency, we proposed to use a coarse-to-fine set of labels for deeply supervising the decoder of our generator G (Fig. 1). This resulted in a multiresolution 1 -distance (MR loss or L M R ) at each decoder layer, as shown in Fig. 1. For {1, . . . , n} layers in decoder, the loss at each resolution P = { p 2 n , p 2 , . . . , p} is given as: whereŷ = G(x) are the generated samples. The MR loss operates on real labels and synthetic representations at all resolutions of the generator decoder. A set of unit kernel (1 × 1 × 1) 3D convolutional operators {M 1 . . . M n } was used to align coarse-to-fine tensors across the channel dimensions. The multiresolution labels for supervision were generated using nearest neighbor interpolation. The loss for our conditional GAN generator can thus be extended as: where the first term in (5) now denotes the least squares adversarial loss minimized by the generator.

B. MEAL: Multiview Perceptual Similarity
Neural style transfer (NST) approaches typically rely on high-dimensional projections of images from large pre-trained models, trained for higher level tasks such as image classification and object detection. The distance between these internal activations, called the learned perceptual image patch similarity (LPIPS), is used to quantify perceptual similarity between two samples [31]. We propose to extend this to multiple views for general applicability to volumetric style transfer, as shown in Fig. 2. To compute LPIPS between two 2D slices s andŝ ∈ R C×H×W , high-dimensional embeddings were extracted from a pre-trained VGG-16 (Fig. 2). These embeddings were then unit-normalized along the channel dimension to get feature stacks E s = {e 1 , e 2 , . . . , e L } and Eŝ = {ê 1 ,ê 2 , . . . ,ê L } corresponding to each slice. The overall LPIPS loss is then calculated by first scaling the 2 -distance between embeddings E andÊ along the channel dimensions by w ∈ R C , followed by spatial and layer-wise averaging, expressed as: We extend this slice-based objective in (6) to multiple views of a medical image volume. For real y and synthetic patchesŷ = G(x) patches, where each y,ŷ ∈ R 1×H×W ×D , we transposed each volume to compute LPIPS across all three dimensionsaxial, coronal and sagittal (shown in Fig. 2). The multiview perceptual similarity (MEAL) between two volumes was thus defined as: where L(·, ·) denoted L LP IP S (·, ·) in (7) and each term corresponded to a different view of the volume. This simple extension enabled style transfer at a volumetric level. The overall cost function minimized by the generator for the Lung2Lung framework is: least squares adversarial loss

C. Context-awareness via self-ensembling
Patch-based generative frameworks often suffer from a lack of global contextual information that is useful for image synthesis. We address this problem by training an ensemble of Lung2Lung frameworks, each conditioned on synthetic outputs from the preceding model, as shown in Fig. 3. Such iterative conditioning helps the successive models to better capture the overall context thereby iteratively improving patch synthesis. Since each model is conditioned on the output from a similar pre-trained predecessor, we call the modeling framework to be self-ensembling -Lung2Lung SE .

D. Generator and Discriminator Architecture
We used the UNet architecture as our generator backbone [32], [35]. Fig. 4 shows the encoder-decoder structure of our network, where each layer from the encoder is symmetrically connected (via skip connections) to the corresponding layer in the decoder. The input to the generator is a patch of size 128 × 128 × 128 with C channels. We set C = 1 for training the self-ensemble Lung2Lung SE model. The output of the generator is a single-channel patch of size 128 × 128 × 128. The structure of each encoder block C and decoder block U, along with the kernel size k and stride length s, is detailed in Fig. 4. All encoder-decoder blocks used 3D convolutions. Several intermediate outputs at multiple spatial resolutions were also output from the generator where 1 × 1 Fig. 4. A modified UNet-like architecture for the generator. The model begins by generating a feature map of size f which is then increased upto 64f at the end of the bottleneck. Output of the model is a single channel patch with same spatial dimensions. Each downsampling block C was composed of 3D convolutions followed by instance normalization (IN) and rectified linear units as activations (ReLU). Similar blocks were used for discriminator model as well. Within the upsample block U the feature maps were first upsampled using trilinear interpolation operator U ↑2× followed by 3D convolutions. For generator f = 24 and discriminator f = 8. × 1 convolutions were used to reduce the channel dimension to 1 for computing the MR loss.
We used a Markovian PatchGAN discriminator in this work [32]. Instead of providing adversarial feedback over the entire image, the PatchGAN discriminator attempts to classify image patches, where the size of each patch is smaller than the overall spatial dimension of the input tensor. Four downsampling blocks (C's shown in Fig. 4), identical to the generator, were used to define the discriminator architecture. For input tensor of size 128 × 128 × 128, we set the patch size to 8 × 8 × 8. Such a patch-based discriminator models the entire image as a Markov random field and helps capture highresolution details present in smaller patches. For optimizing the generator and discriminator parameters, we used Adam optimizer with imbalanced learning rates 0.0002 and 0.00005, respectively. The hyperparameters λ 1 and λ 2 were set to 100 for all experiments. All models were implemented using the open source framework PyTorch [36] and were trained on a single NVIDIA RTX8000 GPU with a batch size of 4.

A. Dataset and Preprocessing
For training and evaluating our models, we utilized data from the SubPopulations and Intermediate Outcome Measures in COPD Study (SPIROMICS) cohort [37]. SPIROMICS is a multi-center, prospective cohort study that acquired breathhold CT scans for every participant at two different lung volumes -total lung capacity (TLC) and residual volume (RV). The original resolution of the chest CT scans was approximately 0.6 × 0.6 × 0.5 mm 3 , and the image size was 512 × 512 per slice, with 500 to 600 slices per volume [38]. SPIROMICS acquired scans from subjects with varying degrees of disease severity, defined by the Global Initiative for Chronic Obstructive Lung Disease (GOLD) [39]. GOLD 1 to GOLD 4 denotes mild to severe COPD, while asymptomatic smokers were grouped in GOLD 0. A small subset of normal individuals who never smoked (NS) was also included for analysis (Table  I). Before training, CT at I RV was deformed into the space of TLC images I T LC using the transformation estimated by a mass preserving DIR method [33]. Image volumes at TLC and RV were then resampled isotropically to a resolution of 1 × 1 × 1 mm 3 . To remove outliers arising due to calcification or metal artifacts, intensity values were clipped to the interval [-1024, 1024] Hounsfield units (HU). The image volumes were then cropped to the bounding box containing the union of the lung regions of the inspiration and expiration. A multi-resolution convolutional neural network was used to segment the lung regions [8]. We further normalized the image intensities between -1 and 1. The disjoint training and testing sets had 1055 and 512 subjects, spanning the GOLD spectrum of disease severity, illustrated in Table I.

B. Clinical and Quantitative Evaluation
We evaluated our models using several quantitative metrics. To assess voxelwise differences between the real images y and synthetic imagesŷ = G θ (x), we used peak signal to noise ratio (PSNR) in decibels (dB), mean absolute error (MAE) in Hounsfield units (HU), and normalized mean squared error (NMSE). All metrics were computed within the lung region defined by mask. PSNR between y andŷ was defined as PSNR(y,ŷ) = 20 log 10 max{y} y−ŷ 2 , where max{y} was the maximum possible value of image intensities, and NMSE was expressed as NMSE(y,ŷ) = y−ŷ 2 2 y 2 2 . To further assess voxelwise consistency, Spearman's correlation co-efficient (r s ) between image intensities within the lung region was also used. Overall perceptual similarity between real and synthetic volumes was quantified by the structural similarity index (SSIM) [40] stated as SSIM(y,ŷ) = (2µyµŷ+c1)(2σ yŷ +c2) (µ 2 y +µ 2 y +c1)(σ 2 y +σ 2 y +c2) , where y andŷ indicated real and synthetic 3D volumes, Fig. 5. Visual comparison of representative samples shown across coronal, axial, and sagittal views from the same subject. We compare results from Lung2Lung with two slice-based methods, Pix2Pix [32] and the work of Yang et al. [11], and a volumetric method Pix2Pix 3D . Both 2D methods present with discontinuities across axial and sagittal view (red), since they were trained on mid-coronal slices, while Pix2Pix 3D produced blurry results (yellow).
respectively. Here, µ's denoted means and σ's denote standard deviations and a window of size 11 × 11 × 11 was used. To further assess agreement between the means of real and synthetically generated images, we conducted a Bland-Altman analysis [41]. We also evaluated sample quality by segmenting pulmonary fissures on the real and synthetic images. In doing so, we sought to evaluate whether the synthetic volumes captured the shape and grayscale characteristics of the fissures reliably. Fissure segmentation was performed using FissureNet, a state-of-the-art lung fissure segmentation method [42].
Most of the studies involved with the development of generative models tend to ignore clinical validation of the synthetic samples generated by their models. We sought to rigorously evaluate the clinical reliability of our samples by computing air trapping and parametric response mapping (PRM)-based emphysema and functional small airways disease (fSAD). These biomarkers, which require matched inspiratory and expiratory image pairs, have been shown to be associated with various disease outcomes in COPD.

C. Comparison with state-of-the-art
We compared performance of Lung2Lung with two state-ofthe-art 2D GAN frameworks -Pix2Pix [32] and the framework by Yang et al. [11]. Pix2Pix framework forms the basis for most of the generative modeling approaches in medical imaging, while the work by Yang et al. [11] was the only study that investigated perceptual loss functions for 2D images denoising. We also compared the performance of Lung2Lung with a commonly used volumetric image synthesis framework Pix2Pix 3D , the 3D counterpart of Pix2Pix [32]. Lastly, an ablation study investigating the impact of MEAL and selfensembling was conducted.

V. RESULTS
We display volumetric synthesis results in multiple crosssectional views in Fig. 5. All 3D models, including Pix2Pix 3D , Lung2Lung, and Lung2Lung SE , generated spatially consistent results. To the contrary, the 2D GAN models (Pix2Pix and Yang et al. [11]), which were trained using coronal slices, produced discontinuities across axial and sagittal views (red regions in Fig. 5). Also, the Pix2Pix 3D generated blurred images with several regions missing important details (see yellow regions in Fig. 5). The Lung2Lung framework, which incorporated MEAL and MR loss, performed significantly better than all other models with PSNR 24.33 dB and SSIM of 0.901. Self-ensembling further improved PSNR from 24.33 dB to 24.53 dB (p < 0.0001), MAE from 88.52 HU to 86.84 HU (p < 0.0001), and Spearman's correlation from 0.619 to 0.631 (p < 0.0001). Wilcoxon's signed ranked test indicated a statistically significant difference in performance between Lung2Lung SE and all other models. Also note that the volumetric methods performed better than the slice-based methods, as shown in Table II. Quantitative performance of Lung2Lung SE across varying levels of disease severity, ranging from GOLD 0 to GOLD 1, is also shown in Table III.
We also conducted a regression analysis for Lung2Lung SE between the means of real I mean RV and syntheticallyÎ mean RV generated RV scans, respectively (Fig. 6). The means showed good agreement with an overall coefficient of determination r 2 = 0.67. A concurrent Bland-Altman analysis, also shown in Fig. 6, showed small variability in error between the means of real and synthetic RV scans. Qualitative performance evaluation of Lung2Lung SE across varying COPD severity demonstrated that the model performed well across all GOLD stages and was able to consistently capture intricate vessel structures, highlighted in yellow (Fig. 7).  II  QUANTITATIVE PERFORMANCE EVALUATION OF LUNG2LUNG ACROSS 512 SUBJECTS FROM THE SPIROMICS COHORT, IN CONTRAST TO VARIOUS  STATE-OF-THE-ART 2D AND 3D GENERATIVE FRAMEWORKS. THE 3D VOLUMETRIC FRAMEWORKS PERFORMED BETTER THAN 2D PLANAR  METHODS, INDICATED BY PSNR, SSIM, MAE (HU), SPEARMAN'S CORRELATION, AND NMSE. WILCOXON SIGNED RANK TEST WAS  USED TO ASSESS DIFFERENCES BETWEEN THE MEANS OF ALL EVALUATION METRICS, WHERE LUNG2LUNG WITH SELF-ENSEMBLING  WAS SELECTED AS REFERENCE. BEFORE PERFORMING THE WILCOXON TEST, WE EVALUATED NORMALITY USING THE  SHAPIRO-WILK TEST,  We conducted model validation against some well-known imaging biomarkers, derived jointly from TLC and RV scans, used to characterize COPD. These included CT density-based air-trapping [43] and image registration-based PRM characterization of functional small airways disease (fSAD) [3]. Percent air trapping was defined as the fraction of voxels below −856 HU within the lung region in the RV scan. For computing fSAD, joint histograms of co-registered TLC-RV image pairs were used. fSAD was defined to be the fraction of voxels between -950 HU and -810 HU in TLC and between -1000 HU and -857 HU in an RV scan. We assessed differences in percent air-trapping and fSAD when computed using the real and synthetic RV scans (Fig. 8). Synthetic RV images generated by Lung2Lung SE yielded similar air trapping and fSAD percentages as compared to real RV scans. Lung2Lung slightly overestimated air trapping and fSAD but Pix2Pix 3D was not able to capture the patterns reliably; see the comparison illustrated in Fig. 8. In Fig. 9, we also show the impact of MEAL on axial slices in comparison to Pix2Pix 3D , which clearly underestimated air trapping to be 13.10%. Lung2Lung estimated 28.77% air trapping for the same subject which was very close to the ground truth value of 27.41%. Fissure segmentations on real and synthetic RV volumes, from pretrained FissureNet, are shown on sagittal slices in Fig. 10.

VI. DISCUSSION AND CONCLUSION
Multiple volume surrogates of lung function have gained widespread clinical attention for characterizing local functional abnormalities in COPD [3]. While these measures enable better understanding of disease mechanisms, they require CT images at different volumes which are not recommended in most clinical settings. We hypothesized that a CT volume at end-inspiration contained sufficient structural information that could be used to predict the associated end-expiratory CT volume. To that end, we proposed a volumetric style transfer framework for translating inspiratory CT to the paired expiratory volume. Although, we demonstrated image translation from a TLC CT to RV, the proposed method can also be used to estimate CT at other lung volumes such as the functional residual capacity. Image-to-image translation for large 3D volumes entailed several challenges. Foremost, was the large GPU memory required to handle increasingly large CT volumes. Most of the existing generative methods address this by training on 2D slices which leads to discontinuous transitions, limiting the radiological utility of these synthetic images. We also observed a decreased performance in 2D models when compared to their 3D counterparts (Table II). We trained our models on 3D sub-volumes or patches, continuous in three dimensions, that were combined at inference by taking the mean along the overlapping boundaries. This helped us estimate a smooth 3D volume by avoiding discontinuities across patches. Patchbased networks are often also limited by the size of receptive field and may not be able to model global patterns within an image volume. We attempted to address this problem by devising several measures. We used relatively larger patches of size 128 × 128 × 128 and increased the overall receptive field by training deeper networks. To further alleviate lack of context, we used coarse-to-fine voxelwise consistency and self-ensembling.  is that they do not attempt to model the inherent style and texture patterns during image translation. Style transfer at a volumetric level is limited due to a lack of 3D models that can be used for generating deep representations. Moreover, stylization typically requires exemplars that are missing for most medical image translation tasks. For our task, modeling texture within the expiratory CT was pertinent since most clinical measures depended largely on variations in local tissue texture. We proposed a multiview perceptual similarity (MEAL) to model image textures in 3D. Adding MEAL significantly improved perceptual image quality when compared to the Pix2Pix 3D , as shown in Fig. 5. Self-ensembling further improved the overall quantitative performance of the model (shown in Table II). An important part of our work was constituted by the clinical validation of our model. Lung2Lung SE performed well across varying GOLD stages and it was easier for the model to capture changes across more severe disease (Table  III). We also highlighted some intricate vessel structures captured successfully by Lung2Lung SE in Fig. 7. When fSAD and air-trapping was computed using real and synthetic RV scans, Lung2Lung SE captured a similar distribution of these biomarkers compared to the ground-truth values (Fig. 8). A visual comparison in Fig. 9 also showed consistency in spatial distribution of air trapping across real and synthetic samples. Identical fissure boundaries identified by an independently trained model showed that our model reliably approximated the real data distribution (Fig. 10). Model validation through an independently trained network demonstrated realism of the synthetic samples.
It is common in clinical practice to acquire lung images only at inspiration, and yet it is becoming evident that an early sign of numerous lung diseases is seen on the expiratory image, manifesting as air trapping or fSAD. Our method allows for the retrospective assessment of patient scans to assess if pathologies were preceded by fSAD, providing critical insights into disease processes. Additionally, the assessment of fSAD is dependent upon a patient making an appropriate effort to expel as much air from the lungs as possible. Furthermore, this difficult maneuver must be repeatable so as to allow for the tracking of disease progression. Lung2Lung eliminates the need for the patient to achieve a breath hold at this low lung volume, thus allowing for the more accurate tracing of longitudinal changes. Our method also allows for the automated reporting of functional abnormalities across a wide variety of lung diseases providing the potential for expanded insights into early pathologies associated with transplant rejection, prediction of rapid progression of emphysema or fibrosis, and more. In summary, our method allows for retrospective evaluation where multiple lung volumes have not been imaged and prospectively where patient cooperation limits the repeatability of an inspiratory/expiratory image pair.