{'fraction_non_alphanumeric': 0.026903749516814842, 'fraction_numerical': 0.006983636129364773, 'mean_word_length': 4.91735285147911, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Monocular Depth Estimation for Semi-Transparent Volume Renderings

Neural networks have shown great success in extracting geometric information from color images. Especially, monocular depth estimation networks are increasingly reliable in real-world scenes. In this work we investigate the applicability of such monocular depth estimation networks to semi-transparent volume rendered images. As depth is notoriously difficult to define in a volumetric scene without clearly defined surfaces, we consider different depth computations that have emerged in practice, and compare state-of-the-art monocular depth estimation approaches for these different interpretations during an evaluation considering different degrees of opacity in the renderings. Additionally, we investigate how these networks can be extended to further obtain color and opacity information, in order to create a layered representation of the scene based on a single color image. This layered representation consists of spatially separated semi-transparent intervals that composite to the original input rendering. In our experiments we show that adaptions of existing approaches to monocular depth estimation perform well on semi-transparent volume renderings, which has several applications in the area of scientific visualization.


INTRODUCTION
Visualization of volumetric data is a common necessity in many applied sciences, to explore new phenomena, understand the acquired measures or to make diagnoses based on them. Methods like direct volume rendering (DVR) are commonly used in practice nowadays, and countless visualizations have been created using DVR or similar techniques. In general, DVR is the preferred method to visualize 3D data using semitransparent structures, allowing for inspection of multiple structures at once that would normally occlude each other. However, such volume renderings usually have to be created with great care and often require a significant amount of manual labor to find good rendering parameters, such as the transfer function (TF), which assigns optical properties to the data. In practice, parameters like the TF, that are essential in order to reproduce such a visualization, are often not shared, or even the underlying original data is unavailable, which makes volume renderings impossible to reproduce or change in a meaningful way.
In this work, we investigate the abilities of neural networks to extract additional relevant information from single RGB images of such visualizations, with the goal of making these rendered images re-usable through composition. As basis for our approach, we investigate existing approaches for monocular depth estimation and adapt them to semi-transparent scenes. For this, a less ambiguous definition of depth is required, as semi-transparent renderings do not have a single depth value as opaque scenes, but rather have multiple relevant structures lying behind each other on a view ray. Looking at prior work, there are some attempts to find the most relevant of these structures in the context of picking (e.g., [1,35]). We employ these techniques to extract depth information for surfaces of interest, and refer to these techniques as depth techniques in the remainder of this paper.
Throughout this paper, we compare the performance of multiple state-of-the-art (SotA) monocular depth estimation networks for different depth techniques and different degrees of opacity in the rendered images. To our knowledge, this is the first study considering monocular depth estimation on volume rendered images. Based on our findings, we modify the SotA models to not only predict the depth of the surface of interest, but also the color and opacity of the semi-transparent structures in front, as well as the color behind this surface of interest (see Fig. 1). Based on the obtained layers, which can be predicted from a single RGB-only volume rendered image, layer-wise re-compositing of the original renderings becomes possible, such that for instance additional meshes can be faithfully integrated into the volume rendering. All without any knowledge about the underlying volume data, transfer function or any other rendering parameter.
To achieve our goals, we make the following contributions within this paper: • We evaluate SotA monocular depth estimators on volume rendered images subject to different amounts of semi-transparency.
• We extend the best performing model to predict a layered image representation, containing the depth of a visually dominant surface, as well as color and alpha layers in front and in the back of these surfaces, enabling the re-composition of the single input color image.
• We demonstrate the efficacy of our approach by compositing mesh geometries into existing volume renderings for which we only have access to RGB images.
We further make our code, datasets and trained models publicly available 1 in order to enable other visualization researchers to use and further extend the presented techniques.

RELATED WORK
Within this section, we discuss prior work related to our approach. We will first provide an overview about depth techniques in the context of direct volume rendering, before we briefly recap the SotA in monocular depth estimation. Volume rendering depth. Traditionally, volume rendered images are generated using the emission absorption model, which is a physicsinspired model underlying the popular volume rendering integral [24]. While this approach leads to faithfully volume rendered images of the volume data at hand, the cloudy nature of the resulting imagery let researchers investigate methods to visually emphasize more well-defined and delineated objects, to support object detection and quantification. The most seminal work in this direction is most likely Levoy's early paper on gradient-based surface extraction from volume data [20]. Kindlmann's and Durkin's build up on this idea, to obtain gradient-based transfer functions [14], in which they exploit the gradient magnitude at a sample in order to modulate the sample's opacity. Further, inspired by this approach, several other transfer function approaches followed, that exploit particular properties to modulate opacity and thus obtain more surface-like representations (e.g. [4,7,15,27]). Naturally, with the opportunities to represent well delineated objects in volume rendered images, the wish to fuse volume rendered images with geometry meshes arose [13]. While this can be directly achieved when extracting surfaces from volumes [26], the integration of geometry into standard ray-casting based volume renderings [16] required the generation of a meaningful depth map in order to support correct image compositing [21]. The most straight forward approach to obtain depth values for a volume rendered image, is the use of FirstHit depth values, where the depth value represents the depth of the first non-transparent sample along a viewing ray. Needless to say, that a single depth value cannot represent the complexity of a volume rendered image containing semi-transparent structures. The same observation is true for maximum gradient magnitude based depth maps, whereby the depth value represents the depth of the sample with the maximum gradient magnitude along a view ray. Other, more advanced approaches allow for the extraction of multiple depth values along a ray. Lindholm for instance applied a-buffer based depth peeling to obtain a complex depth structure, quite faithfully representing the complex nature of the volume rendered object [21]. Bruckner and Gr"oller presented the MIDA method, which is not particularly targeted at extracting depth values, but can obtain meaningful surface representations by exploiting the compositing gradient [3]. Similarly, the what you see is what you pick (WYSIWYP) approach allows for the extraction of several layers, 1 URL added after publication originally meant to support picking of visually salient features [32,35]. While we suggest using such approaches to compute depth maps during volume rendering, Stoppel and Bruckner have also shown that layers can be used to guide interaction widgets in the same context [31]. Apart from these approaches, which are somehow integrated into the rendering process, also other more advanced models exist, which support the extraction of boundaries. Lindholm et al. for instance propose the improved visualization of boundary surfaces by exploiting materialspecific reconstruction in the classification process [22]. Monocular depth estimation. After the first approaches started to leverage convolutional neural networks (CNNs) to directly estimate depth from a single input image [8,9], monocular depth estimation techniques developed in a rapid way. Scores on famous benchmarks such as NYU [30] and KITTI [12] have been surpassed multiple times. While standard CNNs reduce the spatial resolution of feature maps, other work implements multi-layer convolutional networks [17] and multiscale networks [8,9,18]. Framing depth estimation as classification task, Fu. et al. introduce deep ordinal regression networks which divide depth values into discreet ordinal labels [11], instead of directly regressing depth [19,37]. Maximov et al. improve depth estimation using domain invariant defocused synthetic images as supervision in order to close the reality gap [25].
Segmentation masks which separate individual objects in the input image pose a strong prior to depth estimation. Wang et al. perform semantic image segmentation and apply a divide and conquer strategy to depth estimation [33]. Independently, for each segment, depth maps are predicted in a canonical space and then recomposed into global space. Zhu et al. use semantic segmentation masks for better depth estimation, especially along object borders [39]. In the work of Ramamonjisoa et al. they also focus on enhancing depth predictions around occlusion boundaries using displacement fields [28]. Each dataset comes with distinct characteristics and biases. Circumventing domain bound depth predictions, Ranftl et al. combine multiple large datasets in order to train a network that is invariant to changes in depth range and scale [29]. Virtual normals have been introduced by Yin et al. enforcing high-order geometric constraints in 3D space for the depth prediction task [37]. Their geometric loss function projects the predicted depth values into 3D, by using a pinhole camera model, enabling to compute virtual normals, to encode geometric constraints. More recently, Zhao et al. trained a network that removes clutter and novel objects from real images, in order to improve depth prediction [38]. In the work of Lee et al. [18], they present a novel local guidance layer, which is used to compute locally-defined relative depth estimations, on different levels of resolution. While very few monocular depth estimation papers also mention transparent surfaces [6], to our knowledge no systematic study as well as possible applications in the context of volume rendering have been investigated.

METHOD
To investigate SotA monocular depth estimation in the context of volume rendering, we have created large scale training data sets to train existing estimators. Here we first discuss these datasets, how they were generated, and how we controlled the amount of opacity in them. Furthermore, we detail how we tackle the depth ambiguity apparent in semi-transparent structures, resulting from the lack of clear surfaces. Therefore, we compare multiple depth techniques as mentioned above. We then elaborate on the differences to the common real-world datasets for monocular depth estimation and the resulting modifications of the neural models. Lastly, we detail how we extend the monocular depth estimation models to also predict color and opacity layers in addition to depth, allowing for the de-composition of the input RGB-only volume rendering to enable visualization operations, such as for instance geometry integration.

Dataset Properties
The standard monocular depth estimation task requires predicting a depth map from just an RGB color image. Most existing approaches to monocular depth estimation are trained for real-world scenes, with the NYU [30] and KITTI [12] datasets emerging as the two most relevant ones, covering indoor and outdoor scenes. Here, we compare those standard datasets to our requirements when applying the task to images containing semi-transparent structures. Mainly, we have identified the following three differences: Depth sparsity. The standard datasets generally provide quite densely labeled depth maps, with only few missing values due to reflections and occlusions in the ground truth acquisition. For the case of synthetic volume rendered images, we usually have much sparser images, as a large percentage of pixels covers background without a relevant depth. During training, this high percentage of irrelevant pixels would dominate the loss, leading to poor training signals. We tackle this problem using loss masking. Absolute vs. relative depth. Another relevant aspect is the depth range in the datasets. The standard datasets commonly denote their depth labels in meters, whereas synthetic renderings often do not encode real-world units at all, but rather use a rendering-centered scaling of the depth values within the view frustum. As a result, we are more interested in correctly predicting depth values relative to each other, as opposed to producing accurate absolute values. Depth ambiguity. Lastly, depth is quite ambiguous to define in semitransparent renderings, because there are no clear surfaces to choose, as compared to the real-world datasets. In addition to that, there may also be multiple relevant structures in front of each other without full occlusion, allowing for multiple valid choices. In this work, we first focus on predicting the most relevant surface from a visual perspective. For this, we looked at solutions from prior work in the context of picking. A picking layer holds information about what structure or object is "picked" (for each pixel) during a mouse interaction. Naturally, a similar ambiguity problem arises and prior work has proposed methods to decide for the most relevant structure. We make use of this prior work and cover five different depth techniques in Sec. 3.2.

Dataset Generation
As basis for our synthetic dataset, we need a large corpus of volume data for which we can generate semi-transparent renderings automatically in order to create a sufficiently large training dataset. We chose to generate our training dataset from the CQ500 dataset by QureAI [5], which consists of 491 CT scans of human heads. Of those, we used 397 to generate training images, 40 for validation and 40 for testing of the neural network. The volumes have a resolution of 512 × 512 per slice and have between 101 and 645 slices per volume.
From the volumes above, we generate 100K training images from random viewpoints and with randomized transfer functions (TF). The viewpoints are drawn at random from a uniform distribution of viewing directions with a random distance between 2.5 and 3.0 to the volume center. This configuration places the volume around the center of the view frustum. The scenes are rendered with a fixed far plane distance f = 5.0, and we define our depth d as the normalized distance to the camera c for a sample position p: d = |p − c| 2 2 / f OPAQUE: < 10% of pixels influenced by semi-transparency TRANSPARENT: > 20% of pixels influenced by semi-transparency Note that with the proposed strategy we do not generate a very large variety in absolute depth values, as most images show visible structure roughly in the range d ∈ [0.3, 0.7], however as noted before, the actual absolute values are not as relevant for our application, as they are not related to real-world units and can be re-scaled to match the desired scene extents.
In order to vary the appearance, and to control the amount of opacity in the renderings, we propose a transfer function generation scheme, similar to the approach proposed by Engel et al. [10]. This scheme generates piece-wise linear 1D transfer functions using randomly generated trapezoids, representing "peaks". For our datasets, we generate TFs containing between one and three of such peaks, while randomizing their widths (along intensity space) and heights (assigned opacity) to achieve a high variation of renderings with a controllable amount of opacity in the rendered images. Controlling the amount of opacity is relevant to investigate how the amount of semi-transparent structures complicates the monocular depth estimation task in comparison to fully opaque scenes. Hence, we generate two different datasets: Hereby, we classify a pixel as influenced by semi-transparency if its accumulated opacity is in the interval [0.02, 0.9], where an opacity of 1.0 corresponds to fully opaque and 0.0 to fully transparent. For the generation of the TRANSPARENT dataset, we decreased the maximum height of the generated trapezoid peaks compared to OPAQUE from 0.6 to 0.4. During generation, we reject all samples that do not satisfy the amount of opacity condition. The resulting OPAQUE dataset has an average of 5.2% semi-transparent pixels and TRANSPARENT has an average of 29.8% semi-transparent pixels. During training we further apply the standard image augmentation techniques employed by the respective depth-estimation method.

Depth Techniques
Due to the ambiguity of surfaces in semi-transparent renderings, it is notoriously difficult to find a definitive depth value for a given pixel in such renderings. Prior work on volume picking offers a variety of approaches that are designed to find "relevant" surfaces or objects from a human perspective [36]. Together with some common choices, we compare a total of five depth techniques that we include in our experiments. Assuming a ray r(t) = c + t ·⃗ r with ray direction⃗ r and local and accumulated opacities σ r (t) and σ * r (t), we use the following depth techniques (compare Fig. 2

):
FirstHit. This is the simplest strategy. The depth of a pixel is given by the position of the first sample that has a non-zero opacity along a view ray. This strategy completely discards semi-transparency and essentially treats the scene like an opaque object.
MaxOpacity. This strategy corresponds to taking the depth of the sample with the maximum opacity along a ray, i.e., the depth of the sample used for Maximum Intensity Projection (MIP).
MaxGradient. The MaxGradient strategy looks for the steepest change in accumulated opacity along a ray. Similar to the following methods, this favors early samples along the rays.
WYSIWYP. This approach follows the picking strategy presented by Wiebel et al. [35]. This strategy finds intervals along each ray, defined by the zero-crossing of the 2nd order derivative of the accumulated opacity. An interval begins when the 2nd derivative becomes positive after being negative or zero. The interval ends after the 2nd derivative has plateaued at 0 again, corresponding to regions without opacity accumulation. After defining intervals, the chosen depth value is the distance to the start of the interval that increased the accumulated opacity the most.
Here t s , t e denote the start and end depth of the interval, and the first interval begins at t s,i = d FH . The intervals are illustrated on top of Fig. 2. MIDA. Lastly, we took inspiration from Maximum Intensity Difference Accumulation proposed by Bruckner et al. [3]. MIDA is a compositing technique to combine benefits of DVR and MIP. In their work, they define a β parameter that trades some already accumulated opacity along a ray for new, highly relevant (in a MIP sense) structures. To find a depth value, we use the sample with the lowest β parameter, i.e., the sample that increases over the previous MIP (up to this sample) the most.

Monocular Depth Estimation
As a first step, we investigate whether neural nets can predict depth under the above requirements. For this, we compare five SotA approaches that have shown great success on real-world data.
In the following experiments, we use the method of Laina et al. [17] (LAINA), one of the first deep convolutional neuronal networks for monocular depth estimation. Further, we use the well-known deep ordinal regression network (DORN), the first approach formulating depth estimation as classification problem. Additionally, we picked the approaches of Lee et al. [18], a multiscale approach (BTS) and Yin et al. [37] using virual normals to enforce geometric constraints (VNL). Both approaches show strong performance on common depth estimation datasets. Finally, we adobt the network of Ranftl et al. [29](MIDAS), which yields robust depth predictions for a combination of multiple datasets, also being scale and shift invariant. In this task, we train the neural nets to predict a depth map for a semi-transparent rendering from just an RGB color image.
To adapt this task to semi-transparent images, we have to make minor adjustments to the originally proposed neural nets. One of the adjustments is to compensate for the fact that the synthetically generated renderings typically do not encode real-world sizes, but rather encode depth as a normalized distance within an enclosed view frustum. Specifically, we adapt the output activation functions, which are usually rectified linear units (ReLU), to sigmoid activations. This scales the network's output to (0, 1), aligning it with the normalized depth we require. We then trained the networks with their proposed training procedure, including data augmentation, and losses on both the OPAQUE and TRANSPARENT dataset. We report our results in Sec. 4.1.

Monocular Layered Representation Prediction
Since we found that the standard monocular depth estimation task works quite well on semi-transparent data, we further extended the approach, to predict additional information on top of the depth map. Given, that we can predict depths from relevant surfaces in the semitransparent renderings, we now further ask the network to separate the color and opacity of the scene at those depths. That means we let the network predict the accumulated color and opacity in front of the predicted depth, as well as in the back of the predicted depth. In the following, we discuss our choice of the depth technique and depth estimator for this task and how they need to be adapted. Lastly, we present two novel ideas to further improve our results. Choice of depth technique. For these experiments, we chose to use We show depth predictions for the best method on the TRANSPARENT dataset in Fig. 3.
only the BTS [18] network in combination with the WYSIWYP depth. The reason is that depth techniques like MaxOpacity and MaxGradient result in very sparse front layers after separation, while using FirstHit would be meaningless, as front would be 0 by definition. MIDA and WYSIWYP both provide reasonable separation, however they both still coincide with FirstHit depths for many of the pixels. In order to include those thin semi-transparent layers in the front, we adjust the depths to be just behind that structure. While MIDA does not consider the thickness of structures, WYSIWYP readily provides full intervals. To include the first-hit structures, we modify the WYSIWYP approach to use the depth at the interval end, instead of the start, if the first interval would be chosen to represent the depth (compare Fig. 2). Choice of neural net. We chose BTS [18] for our continued experiments for multiple reasons. Firstly, VNL [37] and DORN [11] disqualify for our further adaptations, because they are unable to scale to the amount of output channels necessary. They both actually frame the depth estimation as a classification problem, mapping the classes to a discrete set of possible depths. This is very memory intensive and forbids a drastic increase in output size, as we propose it. From the remaining networks, BTS performs the best in terms of δ 1 on the TRANSPARENT dataset for WYSIWYP, which we consider the most meaningful metric for the task. We can also confirm empirically that BTS produces the best results and its training procedure is quite reliable. Layered representation. We now ask the networks to separate the semi-transparent structures in a front layer, before the predicted depth values, and a back layer, behind the predicted depth, as illustrated in Fig. 1. The front layer can easily be retrieved during ray casting, by exporting the accumulated RGBA buffer when reaching the depth value to separate at. The back can then be computed from the front and the fully composited rendering c full by inversion of the compositing step: The resulting prediction is essentially a layered representation of the rendered scene, that composites to the original input image. This layered representation allows for visualization-centric modifications of the input image, like compositing additional objects into the scene, as also demonstrated in Fig. 1.
In order to predict such layered representation, we modify the last layers of BTS to output a 9-channel image, instead of a single-channel image. The 9 channels consist of: • The depth map, • RGBA of the structures in front of the depth map, and • RGBA of the structures behind the depth map (back).
With the inclusion of color layers in the prediction, we also remove any color-related data augmentation that is part of the depth estimator's training procedure. We refer to the resulting network as baseline (compare Fig. 5). While this baseline suffers from blurry layers and often fails to actually compose to the original image, we introduce the following two learning extensions. Residual image output. In the baseline, we find that predicted front and back layers suffer from blurriness, which we attribute to the decoder of the network. This decoder part is prone to introducing blur, and the skip connections from early layers now have to transport a lot more information compared to when just predicting depth. As a solution to the blurriness, we propose Residual Image Outputs. Instead of directly predicting the front and back RGBA layers, we predict the residual to the input image. Using a tanh-activation we predict how much color and alpha must be removed from or added to the input image, while preserving the input pixel for weakly activated neurons. For the alpha channels, we predict the residuals to the mean of the input color channels, as the input image's alpha is unavailable during inference. The idea is described by the following equation, given neural net F , parameterized by Θ and input image x: Compositing loss. In addition, we introduce a novel, visualizationcentered loss function, which we call Compositing Loss. The idea of the loss is to enforce consistency in the layered image representation, by enforcing the front and back layers to alpha-composite to the original input rendering, as described by the following equation, given color and opacity predictionsĉ andα, number of valid pixels N and ground truth color and opacity c, α: Hereby we consider pixels as valid if they have an α > 0, discarding the pixels showing only background, as they would dominate the loss due to the sparsity of our data. Note that this mask would be unavailable during inference, as we cannot expect to have an alpha channel for the input image. During inference we can instead mask the predicted output using the input color channels.
Overall loss. The overall loss function to train the network for joint depth estimation and color/alpha separation is a weighted sum of the SILog loss [9] on the depth map only, a mean absolute error (L 1 ) on the color and alpha channels, and the compositing loss on the recomposited image, which backpropagates through the color and alpha channels as well: In our experiments we chose λ Depth = 5.0 and λ Comp = 10.0 empirically based on the overall magnitude each of the losses has. With this setting the contribution of L SILog to L total is roughly five times the magnitude of each the L 1, RGBA and L comp .

RESULTS
We have performed several experiments to investigate the monocular depth estimation, the monocular layered representation prediction, as well as the applicability in the wild. The depth estimation experiments evaluate how well neural networks perform on images of semitransparent renderings, specifically we (1) measure the effect of the amount of transparency in the input images, (2) compare the different depth techniques presented in Sec. 3.2 to each other and (3) use the predicted depth maps to composite the input image with additional objects.
Based on the results of those first experiments, we choose a neural net and depth technique for our second set of experiments, investigating the monocular layered representation prediction. In this second set of experiments, we (1) compare the proposed baseline for the separation task with the addition of Residual Image Outputs and the proposed Compositing Loss and (2) use the predicted layered representation to recomposite the original input with additional objects under consideration of alpha-blending. Lastly, we apply our approach to renderings taken as screenshots from the seminal VolumeShop paper [2] to demonstrate its applicability in the wild. Metrics. We measure the performance of the networks using metrics commonly used in the domain of monocular depth estimation, as well as image similarity metrics for the RGBA layers: • δ-metric: The δ 1 , δ 2 , δ 3 metrics measure the percentage of valid predictions within a certain margin of relative error (here 25%): • L 1 -metric: mean absolute error of valid pixels • SSIM [34]: The structured similarity index: Hereby ε i are small constants for numerical stability and µ a , σ 2 a and σ ab are the means, variances and covariances within a local neighborhood for all pixels of a (and b) respectively.
For the δ-metrics and L 1 we again only consider the N valid pixels, indexed by i, that are not part of the background.

Monocular Depth Estimation
Experiment 1. In this experiment, we evaluate the applicability of five SotA monocular depth estimation approaches on the different depth techniques introduced in Sec. 3.2. The five approaches we test are DORN [11], MIDAS [29], LAINA [17], VNL [37] and BTS [18]. We chose this set of networks because they are top performers on the monocular depth estimation leaderboards and have publicly available implementations. The goal of this experiment is to find out if such approaches can be applied to semi-transparent images and how the degree of opacity impacts the neural nets. The quantitative results of this experiment are presented in Table 1. Additionally, Fig. 3 shows the depth predictions of the BTS depth estimator, for each of the depth techniques.
Discussion. The first thing to note from the experiment is that all approaches perform reasonably well on the presented data. Especially for the FirstHit depth technique, we get impressive δ-scores, while the networks perform quite similar on the OPAQUE and TRANSPARENT datasets. In contrast, the remaining depth techniques show a bigger drop in performance when decreasing the amount of opacity. The MaxOpacity depth shows the most severe decrease in δ-scores, and we think this is the case, because it is the least biased towards the front and thus the FirstHit depth. MaxOpacity is completely invariant to the position of structures along the ray, whereas all other techniques take the accumulated opacity into account, which increases faster at the beginning of a ray, because there has not occurred so much absorption yet. Experiment 2. Here, we use the predicted depth maps to composite additional objects onto the input image. This experiment does not consider blending of colors and alpha, only showing the color with the lower depth for each pixel. The results are visualized for two different depth techniques in Fig. 4. Discussion. As shown in Fig. 4, the quality of the composition is highly dependant on the depth technique. The FirstHit approach fails in Fig. 4a, as the semi-transparent structures inside the skull make the skull appear flat at the clipping plane, instead of showing the interior of the skull, as WYSIWYP does (see Fig. 4b).

Monocular Layered Representation Prediction
Based on the previous experiments and the adaptability reasons discussed in Sec. 3.4, we choose BTS in combination with the WYSIWYP depth technique for the upcoming experiments. Experiment 1. This experiment evaluates our proposed extended version of BTS quantitatively. We predict the Front, Back and Depth layers from an input RGB image (see Fig. 1) and compare them with the respective ground truths using SSIM and L 1 . The ablation study for our different variants, with Residual Image Outputs and the Compositing Loss can be seen in Table 2 with the according layered representations illustrated in Fig. 5. Discussion. The resulting layers shown in Fig. 5 Fig. 7, where we compare our approach with naively rendering the additional geometry on top of the input image, as well as with a composition of the ground truth layered representation.
Discussion. As can be seen in Fig. 7c, the naive approach to compositing an RGB image with additional geometry usually does not produce any meaningful result, as the geometry simply occludes the image. Our approach on the other hand manages to correctly occlude the cylinders behind the bones of the fingers. Additionally, the cylinders are partially occluded behind the semi-transparent skin of the hand. Compared to the ground truth, our approach results in a slightly more opaque and darker skin color, but is visually still very close to the ground truth. Fig. 6: Out of distribution re-composited images. Despite our training set only contained human CT data, our method generalizes also to other specimen and even objects.

Application in the Wild
In the previous experiments we evaluated our approach on test datasets that are generally close to the training dataset, in the sense that most of the evaluated images contained either renderings of skulls from the CQ500 dataset that we excluded from training, or other human parts, e.g., the hand CT from Fig. 7. To investigate how our approach generalizes beyond this data, we have applied it to renderings of a salamander as well as the engine block (see Fig. 6). While these images still have been generated with the same volume renderer we have used to generate our training data, the ultimate test is to see how our method generalizes beyond such renderings. Therefore, to evaluate such an applicability in the wild, we run our method on screenshots taken from the seminal Vol-umeShop paper [2]. Those screenshots partially contained labels and zoom ins, used different shading techniques and aspect ratios. We used a simple flood-fill algorithm for a very rough background extraction, removed annotations with deep inpainting [23], and resized the images to 512 × 512 pixels before predicting a layered representation. Finally, we re-composited the input image with additional geometry, before resizing the image to its original resolution. The results are shown in Fig. 8. For the opaque objects, where the predicted depth map is close to the FirstHit depth, we make a slight change to the compositing. If the object is mostly opaque, errors in the Front layer can become visually quite obvious, when the Front is blend over the additional geometry, where it should fully occlude the object. Consequently, we make use of the fact that our layered representation composites to an RGBA version of the input image. Using the original input RGB, the predicted alpha of the re-composition of the layered representation and the predicted depth map, we can perform normal depth-based alpha blending. Discussion. The results for nun-human volumes shown in Fig. 6 indicate, that our approach rather learns properties of volume rendered CT images than the actual geometry, as we can still generate appealing results for other specimen and even objects. As shown in Fig. 8, our approach allows for compositing new geometry into existing volume renderings, by only using a single RGB image extracted as a screenshot without having any access to the renderer. While we have no ground truth for these predictions, it is still quite clear that the quality of the depth maps still leave room for improvement. For example in the depth prediction of the hand, most of the blood vessels are not visible and some of the dark shadows in the split-open head cause holes in the predicted depth map. Despite those limitations our method already enables meaningful re-compositions of visualizations taken from the wild.

CONCLUSIONS AND FUTURE WORK
Within this paper, we have presented the first systematic study investigating the capabilities of deep monocular depth estimation techniques in the context of volume rendered images containing semi-transparent structures. We could show, that SotA monocular depth estimators can be used to predict meaningful depth maps for such images. Motivated by these findings, we have extended the BTS approach, to not only allow for the estimation of depth, but also of a layered image represen- Here we compose the input image with a set of cylinders. The naive approach just renders the cylinders on top of the hand, fully ignoring the geometry and transparency. Using our approach, we can correctly composite the scene from the predicted layered representation, accounting for both occlusion and alpha blending.
tation containing color and opacity. Our extension not only modulates the output of the model, but also facilitates residual learning and exploits a novel compositing loss. With the presented approach, we are able to generate meaningful layered representation of RGB-only volume rendered images, and are thus able to modify them according to visualization-centric tasks, such as for instance the integration of mesh geometry.
In the future, we would like to investigate other tasks which could potentially benefit from our approach. We consider the investigation of novel view synthesis, and subsequently changing the transfer function, just to name a few opportunities. We would also like to more systematically evaluate the influence of the underlying shading model on the depth estimation task. Here we took screenshots from the seminal VolumeShop paper [2], resized them to 512 × 512 and removed the background using a simple floodfill algorithm. Using the input's color and the alpha of the predicted re-composited rendering together with the predicted depth map allows for realistic compositing with additional geometry. Note that for the hand we additionally used deep image inpainting [23] to remove the zoom-in frame.