{'fraction_non_alphanumeric': 0.034547684934242436, 'fraction_numerical': 0.008786694434142584, 'mean_word_length': 4.645675632146884, 'pattern_counts': {'":': 3, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Embedding Convolutional Neural Network for Synthesizing CT Image from T1-Weighted MR Image

Recently, more and more attention is drawn to the field of medical image synthesis across modalities. Among them, the synthesis of computed tomography (CT) image from T1-weighted magnetic resonance (MR) image is of great importance, although the mapping between them is highly complex due to large gaps of appearances of the two modalities. In this work, we aim to tackle this MR-to-CT synthesis by a novel deep embedding convolutional neural network (DECNN). Specifically, we generate the feature maps from MR images, and then transform these feature maps forward through convolutional layers in the network. We can further compute a tentative CT synthesis from the midway of the flow of feature maps, and then embed this tentative CT synthesis back to the feature maps. This embedding operation results in better feature maps, which are further transformed forward in DECNN. After repeat-ing this embedding procedure for several times in the network, we can eventually synthesize a final CT image in the end of the DECNN. We have validated our proposed method on both brain and prostate datasets, by also compar-ing with the state-of-the-art methods. Experimental results suggest that our DECNN (with repeated embedding op-erations) demonstrates its superior performances, in terms of both the perceptive quality of the synthesized CT image and the run-time cost for synthesizing a CT image.


Introduction
Computed tomography (CT) and structural magnetic resonance (MR) images are both important and widely applied in various medical cases, e.g., detecting infarction and tumors in head [1,2], detecting both acute and chronic changes in lung parenchyma [3,4], and imaging complex fractures in extremities [5,6]. Recently, it has become desirable that a CT image can be synthesized from the corresponding MR scan. For example, for positron emission tomography (PET) imaging, the CT image is needed to help attenuation correction (AC) in the modern PET/CT scanners [7].
However, the latest PET/MR scanner lacks of CT acquisition, which may affect the quality of the reconstructed PET image. In this case, the challenge can be greatly alleviated with the CT image synthesized from the T1-weighted MR image.
In this work, we aim to address the problem of synthesizing CT from T1 MR. But these two modalities differ largely regarding their image appearances, which makes this synthesis problem challenging to solve. Examples of the T1weighted MR images and the corresponding CT images are shown in Fig. 1. These images are acquired from the same patients, i.e., (a) for the brain, and (b) for the prostate, respectively. In the MR images, the intensity values of 'air' and 'bone', as pointed by the blue arrows and orange arrows, are both low. However, in the CT images, the 'air' appears dark, while the 'bone' turns to be bright. In general, the intensity mapping between these two modalities of MR and CT is highly complex, encoding both spatial and context information in non-linear mapping.

Fig. 1.
Examples of the pairs of MR and CT data from the same patients: (a) brain, and (b) prostate. The inter-modality mapping is highly complex and non-linear, due to inconsistent appearance mapping in the two modalities, i.e., dark air in both MR and CT, while dark bone in MR but bright bone in CT.
There are several reports in the literature focusing on inter-modality medical image synthesis, i.e., from MR to CT.
These methods can be mainly categorized into the following three classes.
(a) Atlas-based methods. In the atlas-based methods [8][9][10], a set of atlases are prepared in advance, each of which consists of both MR and CT acquisitions. Given a new subject with the MR image only, all atlases are first registered with the new subject by referring to their respective MR images. Then, the resulted deformation fields are used to warp the respective CT images of the atlases to the new subject space, from which the subject CT image can be synthesized through fusion of aligned atlas CT images [11]. Clearly, the performance of the above methods highly depends on the registration accuracy, and the synthesized CT image for the new subject is often fuzzy due to registration error and also inter-atlas difference.
(b) Sparse-coding-based methods. These methods [12,13] usually involve several steps in the respective pipelines.
First, the overlapping patches are extracted from the new subject MR image. These subject MR patches are then encoded by a MR patch dictionary that is built from the linearly aligned MR atlases. The obtained sparse representation coefficients are transferred to the coupled CT patch dictionary (also built from the linearly aligned CT atlases), to fuse the respective CT atlas patches for finally synthesizing the subject CT image. Roy S et al. [14] applied this framework for predicting FLAIR image from T1-and T2-weighted MR images. Similarly, Ye et al. [15] estimates T2-and diffusion-weighted MR images from T1-weighted MR. But one main drawback of these methods is that the estimation is computationally expensive due to the need of sparse coding upon all subject image locations.
(c) Learning-based methods. These methods learn the complex mapping from the local detailed appearances of MR images to those of CT images in the same subjects [16][17][18]. In order to address the issue of expensive computation in sparsity learning based methods, Tri et al. [16] presented an approach to estimate CT image from MR using the structured random forest and the auto-context model. Raviteja et al. [19] proposed an unsupervised approach to maximize both global mutual information and local spatial consistency for inter-modality image synthesis. But, such methods often have to first decompose the whole input MR image into the overlapping patches, and then extract features for each patch. After mapping the MR-related features to the (output) CT image patch, one has to take an additional step to integrate all estimated CT patches for synthesizing the entire CT image. Therefore, these methods still suffer from the complexity caused by the patch-by-patch way.
Recently, the convolutional neural network (CNN) has shown its tremendous popularity and good performance in the computer vision and medical image computing fields. CNN is capable of modeling non-linear mapping between different image spaces, without defining the hand-crafted features. Successful applications can be found by reconstructing high-resolution images from low-resolution images [20], and by enhancing PET signals from the simultaneously acquired structural MR [21]. Han et al. [22] also proposed a deep convolutional neural network method for CT synthesis from MR image, which achieved reasonable performance compared to the atlas-based methods. However, this method can only process a single slice through each forward mapping. To handle the problem of 3D MR-to-CT synthesis, this method had to process multiple slices independently, which could often cause discontinuity and artifacts in the synthesized CT images.
In this paper, we propose a deep embedding convolutional neural network (DECNN) to synthesize CT images from T1-weighted MR images. Concerning the examples in Fig. 1, the mapping from MR to CT can be highly complex, as the appearances of these two modalities vary significantly across spatial locations [23]. This large inter-modality appearance gap challenges the accurate learning of CNN. To this end, we decompose the CNN model into two stages: the transform stage and the reconstruction stage. The transform stage is a collection of the convolutional layers that are responsible for forwarding the feature maps, while the reconstruction stage aims to synthesize the CT image from the transformed feature maps. Besides, we also propose a novel embedding block. The embedding block is able to synthesize the CT image from the tentative feature maps in the CNN. Next, the tentative CT synthesis is embedded with the feature maps, thus the newly embedded feature maps become more related to the CT images and can be further refined by the subsequent layers in the CNN. More importantly, we insert multiple embedding blocks into the transform stage to derive our DECNN accordingly.
It is worth indicating that the introduction of the embedding block has greatly strengthened the inter-modality mapping capability of DECNN regarding MR and CT images. In particular, the tentatively synthesized CT images are embedded into better feature maps, which will be transformed forward to refine the synthesis of CT image. Also, this embedding block contributes to the convergence of the training of this very deep network in a back-propagation form.
Moreover, DECNN allows us to process all test subjects in a very efficient end-to-end way.
Our main contributions can be summarized as follows.
 We propose a very deep network architecture for estimating CT images from MR images directly. The network consists of convolutional and concatenation operations only. It can thus learn an end-to-end mapping between different imaging modalities, without any patch-level pre-or post-processing.
 To better train the deep network and refine the CT synthesis, we propose a novel embedding strategy, to embed the tentatively synthesized CT image into the feature maps and further transform these features maps forward for better estimating the final CT image. This embedding strategy helps back-propagate the gradients in the network, and also make the training of the end-to-end mapping from MR to CT much easier and more effective.
 We carry out experiments on two real datasets, i.e., human brain and prostate datasets. The experimental results show that our method can be flexibly adapted to different applications. Moreover, our method outperforms the state-of-the-art methods, regarding both the accuracy of estimated CT images and the speed of synthesis process.
The rest of this paper is organized as follows. In Section 2, we present the details of our proposed DECNN for estimating CT image from MR image. Then, in in Section 3, we conduct extensive experiments, evaluated with different metrics, on both real brain and prostate datasets. Finally, we conclude this paper in Section 4.

Method
CNN is capable of learning the mapping between different image spaces. We adopt the CNN model similar to [20] for the task of MR-to-CT image synthesis, and then develop our DECNN accordingly. As mentioned above, we decompose the CNN model into two stages, i.e., the transform stage and the reconstruction stage, as also illustrated in Fig. 2(a). The transform stage is used to forward the feature maps (i.e., derived from MR images), such that the CT image can be synthesized in the reconstruction stage. Our proposed DECNN is featured by the embedding block, which embeds the tentative synthesis of the CT image into the feature maps. This embedding block is a key factor of our DECNN for more accurately mapping from T1-weighted MR images to CT images.

Transform Stage and Reconstruction Stage
Transform Stage: The transform stage in CNN consists of several convolutional layers, which derive the feature maps from the input MR images and forward them for later CT synthesis in the reconstruction stage. Denoting the input feature maps to the -th convolutional layer as −1 (corresponding to the output of the early layer, the ( − 1)th layer), we can then compute the output of the -th layer as where and represent the kernels associated with the -th convolutional layer, and ' * ' denotes the convolution operator. We choose the parametric rectified linear unit (PReLU) [24] as the activation function due to its outstanding generalization performance in similar tasks [25,26]. Here, is defined as: where is a learnable parameter.

Reconstruction Stage:
The CNN model ends with the reconstruction stage that synthesizes CT images from the feature maps outputted by the last layer of the transform stage. The reconstruction can be attained by a single convolutional layer by following = * + . (3) Here, is the feature map outputted by the transform stage, and is the synthesis estimated by the reconstruction stage. Note that there is no activation function employed in the reconstruction stage. We also require the size of to be equal with the size of the initial input to the transform stage.
The training of the convolutional layers is conducted in a patch-by-patch way in order to alleviate the demand of many subjects for training. Given a center location in a training subject with his/her carefully aligned MR and CT images, Note that a single convolutional layer is needed in the reconstruction stage. We particularly adopt 3 kernels of the support 3 × 3 × 128 to complete the reconstruction. The output of each kernel is associated with a certain 64 × 64 axial slice in the synthesized CT patch. Note that the 64 × 64 axial size is only required upon the training patches. In the testing phase, the network can directly apply convolutional operations upon 3 consecutive axial MR planes of arbitrary size. Thus, the entire CT field-of-view can be synthesized efficiently by the end-to-end mapping of CNN.

Embedding Block and DECNN
The embedding block estimates the CT synthesis given the latest feature maps, and then embeds the tentatively synthesized CT image into better feature maps. The architecture of the embedding block is shown in Fig. 2(b), as stated before. The embedding block consists of 4 steps.
(1) The CT image corresponding to the input MR image is tentatively synthesized from the feature maps that are input to the embedding block under consideration. The synthesis is accomplished by applying Eq. (3) to the input feature maps; this step is the same as the reconstruction stage.
(2) The input feature maps are transformed through a single convolutional layer in the embedding block. The number of the kernels and the support of each kernel are the same as the convolutional layers in the transform stage, which has been described in Section 2.1.
(3) The tentatively synthesized CT image is perceived as 3 additional feature maps, and then concatenated with those (4) These 131 concatenated feature maps are transformed through a subsequent convolutional layer. This layer reduces the number of the feature maps from 131 to 128, implying that there are 128 kernels of the support 3 × 3 × 131 in this layer.
In general, the feature maps are processed through two convolutional layers and one concatenation layer in the embedding block. The sizes of the input and the output feature maps for the embedding block are identical. Thus, the embedding block can be seamlessly integrated into the aforementioned CNN model.
We insert several embedding blocks in the late part of the transform stage, and thus derive our proposed DECNN model as shown in Fig. 2(c). Note that, after each embedding block, an additional convolutional layer is appended to refine the feature maps. Finally, the feature maps arrive at the reconstruction stage, where the CT image is eventually synthesized. In general, with embedding blocks, the total number of the convolutional layers in DECNN will be 3 + 5.
Note that the embedding blocks are essentially important to tackle the highly complex mapping from MR to CT. The deep network with many convolutional layers transforms feature maps forward. The representations captured by shallow layers are more related with the input MR appearances, while the representations captured by deep layers are more related with the CT appearances. The embedding blocks in the late part of the transform stage can first synthesize CT images tentatively, and then inject the tentative CT estimation into the feature maps for refined CT synthesis. In this way, these embedding blocks help fill the large appearance gap between the two modalities and also speeds up the MR-to-CT mapping within the flow of feature maps. The CT synthesis can also be refined across individual embedding blocks. Therefore, the learning of DECNN tends to yield more accurate and effective modeling for the mapping from MR to CT.

Implementation Details
The introduction of the embedding blocks also allows us to compute the auxiliary loss, which helps improve the robustness of DECNN against the vanishing gradients and also strengthens the learning of the network parameters during back-propagation. For the -th embedding block in DECNN, the tentatively synthesized CT image is denoted by .
Concerning the ground truth ̃, we can compute the overall loss function of DECNN as Here, encodes all parameters of the network, and ( ) is the 2 -norm regularization upon the parameters. We set to 0.5 in our experiments, which balances the losses computed from the tentative CT estimation of individual embedding blocks and also the output of the final reconstruction stage.
We implement the CNN-based learning by CAFFE [27] in this work. The kernels in the convolutional layers are initialized following [28]. Similar to [25,29], we set the learning rate to 10 −5 , and adopt "Adam" as the solver for the optimization during back-propagation. The momentum is set to 0.9, and the 2 -norm regularization upon the learned parameters is set to 10 −3 . In DECNN, we transform the input feature maps through 5 convolutional layers empirically, such that the feature maps are ready for the first embedding block to synthesize CT images tentatively. More embedding blocks can be flexibly appended, according to the need in different applications, as described in Section 3 below.
Our network processes the input MR patches covering 3 axial slices, and estimates the output CT patches of the same size. In the training, the axial size is fixed to 64 × 64 for convenient implementation. In the testing, the 3 consecutive axial slices, covering the entire slices, can be processed as a whole. Therefore, each axial slice of the subject CT image can be reconstructed 3 times, which will be averaged for the final CT synthesis. Note that more axial slices can be considered simultaneously, although the learning of DECNN may become more challenging. Note, with the 3 axial slices consecutively synthesized, the superior-inferior continuity of the synthesized CT image can be preserved, which will be confirmed by experimental results in Section 3.5.

Experimental Result
In this section, we evaluate the performance of our method on two real CT datasets, i.e., 1) brain dataset and 2) prostate dataset. We first describe the datasets used for training and testing our method. Next, more detailed training setup is given. Subsequently, we analyze the effect of the embedding blocks in our architecture. We also present both qualitative and quantitative comparisons between our DECNN model and state-of-the-art methods. After that, we show the superiority of our designed quasi-3D mapping in keeping the appearance consistency in each dimension. Finally, we analyze the learned feature maps in our model, to explain why our DECNN model can iteratively get better synthesized CT results.

Datasets
We apply DECNN to the two real datasets for demonstrating its capability. The two datasets are introduced below.
(1) Brain dataset. The brain dataset consists of 16  MR and CT images from the brain dataset and the prostate dataset are shown in Fig. 1.

Experimental Setting
Various studies have found that, with more data, the performance of a deep model can get better [20,30]. So, in order to make full use of the dataset, we adopt a data augmentation strategy similar to [31,32]. All images are left-right flipped. In this way, we are able to have more available images for training the deep model in each leave-one-out case.
Instead of using large-size image for training, patches are generated by dividing each image into a regular grid of small overlapping patches. Specifically, we extract patches of the size 64 × 64 from the pre-processed MR and the ground-truth CT with the stride of 8. Accordingly, we can get enough samples for training a very deep architecture.
To quantitatively evaluate the synthesis performance, we use two popular metrics: 1) mean absolute error (MAE) and 2) peak signal-noise ratio (PSNR). The PSNR is computed as where is the number of voxels in each image, and is the maximal intensity value of and ̃. In general, a higher PSNR indicates better perceptive quality of the synthesized CT image.

Role of Embedding Block
It is necessary to verify the importance of the embedding blocks in DECNN. The embedding blocks can be flexibly appended to the transform stage. With 5 simple convolutional layers prior to the first embedding block, we show the PSNR changes with respect to different numbers of embedding blocks in Fig. 3(a) and (b), when applied to the brain and prostate datasets, respectively. Specifically, "Ebd " indicates embedding blocks used in DECNN, where the overall number of the convolutional layers for transforming feature maps is (3 + 5) as in Fig. 2(c).
From Fig. 3(a) and (b), we can observe that the PSNR measures increase quickly to the best performances with more embedding blocks and thus deeper networks. However, the performance starts to drop once the DECNN model becomes too deep to be well trained, partially due to the small size of each dataset. Specifically, the highest average PSNR metric occurs at "Ebd 2" on the brain dataset, while at "Ebd 4" on the prostate dataset, respectively. The use of less embedding blocks for the brain dataset is mainly because the brain CT images are relatively easy to synthesize.
On the other hand, a deeper network (with more embedding blocks) is needed to map from MR to CT, due to more pelvic anatomical details in the prostate CT images.
The DECNN model is further compared to the CNN model without embedding of the feature maps. Since we apply our proposed method with "Ebd 2" to the brain dataset, we compare it to the CNN model with the same number of convolutional layers (i.e., 11 layers in the transform stage), which is referred to as "11-layer CNN" in Fig. 3 The use of embedding blocks is also helpful for easier convergence of the network, which is fundamentally important when training the DECNN model for practical usage. In Fig. 3(c) and (d), we show the testing performance changes in PSNR, when training the network with more epochs of patches. On both brain and prostate datasets, DECNN converges faster than the CNN model without embedding, even using the same number of the convolutional layers.
In particular, while the depth becomes deeper to handle for the prostate images than the brain images, DECNN leads to a wider margin ahead of the CNN model without embedding. All the observations above imply that the use of embedding strategy upon the feature maps is effective in the optimization of parameters in the convolutional network, especially when the deep network is used to encode the highly complex end-to-end mapping.  5). Also, the changes of the testing performance with respect to the increasing epochs of the training samples are plotted in (c) and (d) for brain and prostate datasets, respectively, to demonstrate the convergence of the training of DECNN. The comparison methods are referred to as the CNN models without embedding, but with the same respective numbers of convolutional layers, such as "11-layer CNN' for the brain dataset and "17-layer CNN" for the prostate dataset.
The CT estimation results by using different 'Ebd i' are shown in Fig. 4 for brain dataset, and Fig. 5 for prostate dataset. In Fig 4, the first and the second columns show the MR image input to the DECNN and the ground-truth CT image, respectively, in the axial, sagittal and coronal views. 'Ebd 0' can estimate the contour of the bone, but also generate many artifacts in the synthesized CT image. With the use of one embedding block, the result of 'Ebd 1' becomes better. When using 'Ebd 2', we can get the most satisfying result compared to the ground-truth. Similar observations can also be found for the prostate dataset in Fig. 5. Note that more embedding blocks are needed for synthesizing the prostate CT images, as the MR-to-CT mapping is more complex for the pelvic region.

Comparisons with State-of-the-Art Methods
In this section, we conduct comprehensive comparisons between our DECNN and state-of-the-art methods, including the atlas-based method ("Atlas") [11], the sparse representation method ("SR") [33], and the random forest with the auto-context model ("RF+ACM") [16]. The results of the CNN models without embedding, which are designated by "CNN" for short, are also summarized.
 "Atlas": When testing a subject, all other training images are used as atlases; that is, we are using leave-one-out cross validation here. To synthesize the CT image of a test subject, the MR image of each training atlas is first aligned to the subject MR image. The deformation field will be used to warp the CT image of each training atlas.
Then, we can have the final CT synthesis by fusing the warped CT images of all training atlases.
 "SR": When all the training atlases are warped to the subject MR image, a non-local sparse representation is carried out. That is, a small MR patch in the subject MR image is represented as a sparse liner combination of non-local MR patches from the aligned training atlas MR images. Then, the representation coefficients are applied to the respective aligned training atlas CT images to obtain the final synthesized CT.
 "RF+ACM": This is a learning-based method which uses the structured random forest to directly estimate a CT image as a structured output from the input MR image. Each leaf node of the trees in the random forest yields an estimate, which can be integrated for the final output. Moreover, the auto-context model is used for further refining the synthesized CT image.
The detailed PSNR measures are shown in Table 1 and Table 2, which correspond to the brain and prostate datasets, respectively. DECNN and CNN clearly outperform the other three methods under comparison, which imply the powerfulness of deep convolutional networks in mapping from MR to CT. Further, with the same depth (or the number of convolutional layers), the proposed DECNN is superior to the CNN model without embedding, by producing higher PSNRs. In general, our DECNN is able to attain the highest performance among state-of-the-art methods.
Visual comparisons for different methods are also provided in Fig. 6 for brain dataset and Fig. 7 for prostate dataset.
For each dataset, we give two exemplar subjects. We can observe that the CT images synthesized by DECNN are more similar to the ground truth in both datasets. The structural similarity is high, while more anatomical details are preserved in the results of DECNN. The CT images synthesized by "Atlas" and "RF+ACM", however, are relatively fuzzy (c.f. the bone boundaries highlighted in the figure). Therefore, the proposed DECNN method yields better perceptive quality of the synthesized CT images.
DECNN is featured by the end-to-end mapping capability that allows us to process each test subject very efficiently.
On the contrary, the conventional methods are usually time-consuming. For example, "RF+ACM" synthesizes the CT images by extracting features first and then processing through random forest regressors with auto-context. Even though the PSNRs of "RF+ACM" are relatively higher than both "Atlas" and "SR" method, its computation efficiency still falls short significantly compared to "CNN" and "DECNN". Specifically, "RF+ACM" takes about 20 minutes to synthesize a CT image for the test subject. With "DECNN", we can complete the computation of a brain subject (of the size 234 × 181 × 149) in 46 seconds, and a prostate subject (of the size 153 × 193 × 50) in 28 seconds. The high speed performance provides the potentials for clinic applications of our method. The above experiments are carried out on a computer with an Intel Core i7 4.00GHz processor, 16 GB RAM, and an NVIDIA GeForce GTX Titan X GPU.

Quasi-3D Mapping
In DECNN, we implement the input as 3 consecutive axial slices from MR. The 3 MR slices are jointly considered as part of the feature maps since the first convolutional layer in the network. The output consists of 3 axial slices that correspond to the input. We synthesize every 3 consecutive axial slices for the subject CT image, and combine the outputs into the final result by simple averaging. We regard this joint synthesis of the 3 consecutive slices as quasi-3D mapping. Examples of the synthesized brain CT images are denoted by "×3 slices" for the implemented DECNN, and compared with the cases of fewer ("×1 slice") or more ("×5 slices") input slice(s) in Fig. 8.
The comparisons are two-fold. First, concerning the quality within the axial plane (c.f. the red boxes in the first row of the figure), "×3 slices" is comparable to "×5 slices" and better than "×1 slice". Second, the outputs of "×3 slices" and "×5 slices" are relatively continuous in the superior-inferior direction (c.f. both the yellow and the purple boxes highlighted on the sagittal and the coronal views). This implies that, with more than one axial slices as input, DECNN models the MR-to-CT mapping in the quasi-3D way. The mapping benefits not only the intra-slice quality, but also the inter-slice continuity. Fewer artifacts are observable within and between the axial planes. We note that, with "×5 slices", the outputs are mostly comparable to "×3 slices" visually (as shown in Fig. 8) and quantitatively (in terms of average PSNR: 27.4 dB for "×5 slices" and 27.3 dB for "×3 slices" on the brain dataset). Accordingly, we adopt the "×3 slices" setting for the proposed DECNN, to reduce the complexity of the computation.  8. The synthesized CT brain images using 1, 3 and 5 axial slices as inputs to DECNN, which are designated as "×1 slice", "×3 slices" and "×5 slices", respectively. The results in the axial, sagittal and coronal views are presented.

Evolution of Feature Maps
For further understanding the mapping from MR image to the corresponding CT image, we analyze the evolving feature maps after several convolutional layers in DECNN. The exemplar feature maps are selected for visualization in Fig. 9 for the brain dataset, and Fig. 9 for the prostate dataset. For the brain dataset, in addition to the 3 consecutive MR slices as input and the 3 CT slices as ground-truth output in Fig. 9 (a), we also show the feature maps produced by individual layers of the adopted 'Ebd 2' model. In particular, the layers of 'Conv 1', 'Conv 3', 'Conv 5', 'Conv 8' and 'Conv 11' are selected for visualization, where 'Conv 1-5' indicate the convolutional layers prior to the first embedding block, and 'Conv 8' and 'Conv 11' are in the end of the two embedding blocks, respectively.
In 'Conv 1', the original MR slices are transformed to a series of feature maps, each of which aims to synthesize brain CT image. That is, the soft tissue structures are discarded while the bone regions of the brain boundaries are focused.
Note, the original MR slices are decomposed into many feature maps according to intensity values, and the feature maps with very dark bone regions will be preserved as they are associated with the bone regions of the brain CT image (the last few columns in 'Conv1' of Fig. 9 (b)). But they are just the shape and location which are right for predicting CT image, and their intensity values are not changed compared to the input MR. So the latter layers are mainly to find the rest mapping on the corresponding region. Specially, in 'Conv 3', the feature map as indicated by the green box becomes white in the bone regions. This verifies the transform from MR space to CT space. In the end of transform block (Conv 5), we can see more CT-like feature maps (as indicated by the blue box). After applying the embedding block, the CT-related feature maps are getting much better (as indicated by both yellow box and red box). Specifically, the feature map in the red box is more satisfying, since there is less noise in the middle of the feature map and all the information in this feature map is more related to the CT image.
Similar observations can be also found on the prostate dataset in Fig. 10 (a-b). But, there are a few differences compared to the brain images, i.e., the decomposition way in earlier convolutional layers, as well as using more embedding blocks for prostate such as using the 'Ebd 4 (proposed)' model. For the case of brain image, the network decomposes MR image according to its intensity value while ignoring structure information in the middle of the brain. However, for the case of prostate CT image, there is also complex boundary information between different soft tissues in the CT image. Therefore, all the decomposed feature maps preserve boundary information from the MR image (as shown in the first row 'Conv 1' in Fig. 10 (a)). Also, prostate CT image has more structural details compared to brain CT image.
Therefore, it needs more embedding blocks to complete the MR-to-CT mapping. Similarly, feature maps in the latter convolutional layers are getting more related to CT image, and details are becoming clearer compared to earlier layers.
For example, both feature maps in the blue box of row 'Conv 14' and in the red box of row 'Conv 17' are related to the bone structures, while the boundary in 'Conv 17' is much clearer than the one in 'Conv 14', which can yield more accurate CT image prediction results. Fig. 9. Example feature maps in each convolutional layer for the case of synthesizing brain CT image.

Conclusion and Discussion
In this paper, we propose a novel DECNN model to synthesize the CT image from the T1-weighted MR image. Deep learning is well known for its capability in encoding the highly complex mapping between two different image spaces.
The embedding block, which embeds the tentative CT estimation into the flow of the feature maps in deep learning, is integrated with CNN in our work. Thus, we derive DECNN that can transform the embedded feature maps forward and reconstruct better CT synthesis results in the end. Our experiments also prove that our proposed embedding block can effectively contribute to more accurate modeling of the MR-to-CT mapping, as well as faster convergence of the training of the deep network. Meanwhile, the overall performance of DECNN, measured by both the quality of the synthesized CT images and the running time, is superior to the state-of-the-art methods.
Note, our work provides a generalized deep learning solution that is not restricted to the MR-to-CT synthesis only.
Moreover, we adopt the end-to-end mapping, such that the resulted CT image can be exactly the same size as the input MR image. In this way, the architecture of our DECNN ensures high efficiency to process the test images. In the future, we will further investigate the performance of DECNN when applied to other image synthesis tasks, thus promoting more usage in the real clinical applications.