{'fraction_non_alphanumeric': 0.03976717840540646, 'fraction_numerical': 0.01648501894605258, 'mean_word_length': 4.602976581308821, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Depth Map Decomposition for Monocular Depth Estimation

We propose a novel algorithm for monocular depth estimation that decomposes a metric depth map into a normalized depth map and scale features. The proposed network is composed of a shared encoder and three decoders, called G-Net, N-Net, and M-Net, which estimate gradient maps, a normalized depth map, and a metric depth map, respectively. M-Net learns to estimate metric depths more accurately using relative depth features extracted by G-Net and N-Net. The proposed algorithm has the advantage that it can use datasets without metric depth labels to improve the performance of metric depth estimation. Experimental results on various datasets demonstrate that the proposed algorithm not only provides competitive performance to state-of-the-art algorithms but also yields acceptable results even when only a small amount of metric depth data is available for its training.


Introduction
Monocular depth estimation is a task to predict a pixel-wise depth map from a single image to understand the 3D geometry of a scene. The distance from a scene point to the camera provides essential information in various applications, including 2D-to-3D image/video conversion [52], augmented reality [35], autonomous driving [8], surveillance [22], and 3D CAD model generation [20]. Since only a single camera is available in many applications, monocular depth estimation, which infers the 3D information of a scene without additional equipment, has become an important research topic.
Recently, learning-based monocular depth estimators using convolutional neural networks (CNNs) have shown significant performance improvements, overcoming the intrinsic ill-posedness of monocular depth estimation by exploiting a huge amount of training data [1,3,6,7,14,25,28,54,56]. Existing learning-based monocular depth estimators can be classified into two categories according to arXiv:2208.10762v1 [cs.CV] 23 Aug 2022 the properties of estimated depth maps: relative depth estimation and metric depth estimation. Relative depth estimation predicts the relative depth order among pixels [2,34,51,58]. Metric depth estimation, on the other hand, predicts the absolute distance of each scene point from the camera [3,6,14,25,54], which is a pixel-wise regression problem.
To estimate a metric depth map, a network should learn both the 3D geometry of the scene and the camera parameters. This implies that a metric depth estimator should be trained with a dataset obtained by a specific camera. In contrast, a relative depth estimator can be trained with heterogeneous datasets, e.g., disparity maps from stereo image pairs or even manually labeled pixel pairs. Thus, relative depth estimation is an easier task than metric depth estimation is. Moreover, note that the geometry of a scene can be easily estimated when extra cues are available. For example, depth completion [38,39,55], which recovers a dense depth map from sparse depth measurements, can be performed more accurately and more reliably than monocular depth estimation is. Based on these observations, metric depth estimation algorithms using relative depths as extra cues have been developed via fitting [34,43] or fine-tuning [42].
In this paper, we propose a monocular metric depth estimator that decomposes a metric depth map into a normalized depth map and scale features. As illustrated in Fig. 1, a normalized depth map contains relative depth information, and it is less sensitive to scale variations or camera parameters than a metric depth map is. The proposed algorithm consists of a single shared encoder and three decoders, G-Net, N-Net, and M-Net, which estimate gradient maps, a normalized depth map, and a metric depth map, respectively. M-Net learns to estimate metric depth maps using relative depth features extracted by G-Net and N-Net. To this end, we progressively transfer features from G-Net to N-Net and then from N-Net to M-Net. In addition, we develop the mean depth residual (MDR) block for M-Net to utilize N-Net features more effectively. Because the proposed algorithm learns to estimate metric depths by exploiting gradient maps and relative depths, additional datasets containing only relative depths can be used to improve the metric depth estimation performance further. Experimental results show that the proposed algorithm is competitive with state-of-the-art metric depth estimators, even when it is trained with a smaller metric depth dataset. This paper has the following contributions: -We propose a novel monocular depth estimator, which decomposes a metric depth map into a normalized depth map and relative depth features and then exploits those relative features to improve the metric depth estimation performance. -The proposed algorithm can be adapted to a new camera efficiently since it can be trained with a small metric depth dataset together with cameraindependent relative depth datasets. -The proposed algorithm provides competitive performance to conventional state-of-the-art metric depth estimators and can improve the performance further through joint training using multiple datasets.

Monocular Metric Depth Estimation
The objective of monocular metric depth estimation is to predict pixel-wise absolute distances of a scene from a camera using a single image. Since different 3D scenes can be projected onto the same 2D image, monocular depth estimation is ill-posed. Nevertheless, active research has been conducted due to its practical importance. To infer depths, early approaches made prior assumptions on scenes, e.g. box blocks [9], planar regions [44], or particular layout of objects [10]. However, they may provide implausible results, especially in regions with ambiguous colors or small objects. With recent advances in deep learning, CNN techniques for monocular depth estimation have been developed, yielding excellent performance. Many attempts have been made to find better network architecture [3,6,14,25,54] or to design more effective loss functions [2,5,16,25]. It has been also demonstrated that the depth estimation performance can be improved by predicting quantized depths through ordinal regression [7], by employing Fourier domain analysis [26], by enforcing geometric constraints of virtual normals [56], or by reweighting multiple loss functions [28]. Recently, the vision transformer [4] was employed for monocular depth estimation [1], improving the performance significantly.

Relative Depth Estimation
The objective of relative depth estimation is to learn the pairwise depth order [58] or the rank of pixels [2,51] in an image. Recently, listwise ranking, instead of pairwise ranking, was considered for relative depth estimation [34]. Also, scaleinvariant loss [6] and its variants [32,33,43,48] have been used to alleviate the scale ambiguity of depths, thereby improving the performance of relative depth estimation.
Unlike metric depths, relative depth information -or depth order information -is invariant to camera parameters. Therefore, even though a training set is composed of images captured by different cameras, it does not affect the performance of relative depth estimation adversely. Therefore, heterogeneous training data, such as disparity maps from stereo image pairs [48,50,51] or videos [43], structure-from-motion reconstruction [32,33], and ordinal labels [2], have been used to train relative depth estimators.

Relative vs. Metric Depths
A metric depth map contains relative depth information, whereas relative depth information is not sufficient for reconstructing a metric depth map. However, relative-to-metric depth conversion has been attempted by fitting relative depths to metric depths [34,43] or by fine-tuning a relative depth estimator for metric depth estimation [42].
On the other hand, relative and metric depths can be jointly estimated to exploit their correlation and to eventually improve the performance of metric depth estimation. To this end, ordinal labels are used with a ranking loss in [2]. Also, in [27], relative and metric depth maps at various scales are first estimated and then optimally combined to yield a final metric depth map.
The proposed algorithm also estimates relative depth information, in addition to metric depths, to improve the performance of metric depth estimation. However, differently from [2,27], the proposed algorithm better exploits the correlation between relative and metric depths by decomposing a metric depth map. Furthermore, the proposed algorithm can provide promising results even with a small metric depth dataset by exploiting a relative depth dataset additionally. Fig. 2 is an overview of the proposed algorithm, which consists of a shared encoder and three decoders -G-Net, N-Net, and M-Net. The shared encoder extracts common features that are fed into the three decoders. Then, G-Net predicts horizontal and vertical gradients of depths, while N-Net and M-Net estimate a normalized depth map and a metric depth map, respectively. Note that features extracted by G-Net are fed into N-Net to convey edge information, and those by N-Net are, in turn, fed into M-Net to provide relative depth features. Finally, via the MDR block, M-Net exploits the relative depth features to estimate a metric depth map more accurately.

Metric Depth Decomposition
Given an RGB image I ∈ R h×w×3 , the objective is to estimate a metric depth map M ∈ R h×w . However, this is ill-posed because different scenes with different metric depths can be projected onto the same image. Moreover, scale features of depths are hard to estimate from the color information only since they also depend on the camera parameters. To address this issue, we decompose a metric  depth map M into a normalized depth map N and scale parameters. The normalized depth map N contains relative depth information, so it is less sensitive to scale variations or camera parameters than the metric depth map M is. There are several design choices for normalizing a metric depth map, including min-max normalization or ranking-based normalization [11]. However, the min-max normalization is sensitive to outliers, and the ranking-based normalization is unreliable in areas with homogeneous depths, such as walls and floors. Instead, we normalize a metric depth map using the z-score normalization. Given a metric depth map M , we obtain the normalized depth map N by where µ M and σ M , respectively, denote the mean and standard deviation of metric depths in M . Also, U is the unit matrix whose all elements are 1. N-Net, denoted by f N , estimates this normalized depth map, and its estimate is denoted byN . When the scale parameters µ M and σ M are known, the metric depth map M can be reconstructed bŷ In practice, µ M and σ M are unknown. Conventional methods in [34,42,43] obtain fixed σ M and µ M for all images based on the least-squares criterion. In such a case, the accuracy ofM direct in (2) greatly depends on the accuracy of σ M and µ M . In this work, instead of the direct conversion in (2), we estimate the metric depth map by employing the features ψ N (I), which are extracted by the proposed M-Net, f N , during the estimation ofN . In other words, the proposed M-Net, f M , estimates the metric depth map bŷ For metric depth estimation, structural data (e.g. surface normals or segmentation maps) have been adopted as additional cues [5,29,37,40], or relative depth features have been used indirectly via loss functions (e.g. pairwise ranking loss [2] or scale-invariant loss [1,6,30]). In contrast, we convert a metric depth map to a normalized depth map. Then, the proposed N-Net estimates the normalized depth map to extract the features ψ N , containing relative depth information. Then, the proposed M-Net uses ψ N for effective metric depth estimation.
Similarly, we further decompose the normalized depth map N into more elementary data: horizontal and vertical gradients. The horizontal gradient map G x is given by where ∇ x is the partial derivative operator computing the differences between horizontally adjacent pixels. The vertical gradient map G y is obtained similarly. The proposed G-Net is trained to estimate these gradient maps G x and G y . Hence, G-Net learns edge information in a scene, and its features ψ G are useful for inferring the normalized depth map. Therefore, similar to (3), N-Net estimates the normalized depth map viaN using the gradient features ψ G (I).

Network Architecture
For the shared encoder in Fig. 2, we adopt EfficientNet-B5 [47] as the backbone network. G-Net and N-Net have an identical structure, consisting of five upsampling blocks. However, G-Net outputs two channels for two gradient maps G x and G y , while N-Net yields a single channel for a normalized depth map N . M-Net also has a similar structure, except for the MDR block, which will be detailed in Section 3.3. MDR predicts the mean µ M of M separately, which is added back at the end of M-Net. The encoder features are fed into the three decoders via skip-connections [13], as shown in Fig. 2(a). To differentiate the encoder features for the different decoders, we apply 1 × 1 convolution to the encoder features before feeding them to each decoder. Also, we apply the channel attention [15] before each skip-connection to each decoder.
We transfer features unidirectionally from G-Net to N-Net and also from N-Net to M-Net to exploit low-level features for the estimation of high-level data. To this end, we fuse features through element-wise addition before each of the first four upsampling blocks in N-Net and M-Net, as shown in Fig. 2 Specifically, let ψ out G and ψ out N denote the output features of G-Net and N-Net, respectively. Then, the input feature ψ in N to the next layer of N-Net is given by where ω G and ω N are pre-defined weights for ψ out G and ψ out N to control the relative contributions of the two features. For M-Net, the features from N-Net are added similarly. In order to fuse features, we use addition, instead of multiplication or concatenation, for computational efficiency.

MDR Block
We design the MDR block to utilize the features ψ N of N-Net more effectively for the metric depth estimation in M-Net. Fig. 3 shows the structure of the MDR block, which applies patchwise attention to an input feature map and estimates the mean µ M of M separately using the transformer encoder [4]. Note that the transformer architecture enables us to use one of the patchwise-attended feature vectors to regress µ M .
Specifically, MDR first processes input features using an 8 × 8 convolution layer with a stride of 8 and a 3 × 3 convolution layer, respectively. The patchwise output of the 8 × 8 convolution is added to the positional encodings and then input to the transformer encoder [4]. The positional encodings are learnable parameters, randomly initialized at training. Then, the transformer encoder generates 192 patchwise-attended feature vectors of 128 dimensions. We adopt the mini-ViT architecture [1] for the transformer encoder. The first vector is fed to the regression module, composed of three fully-connected layers, to yield µ M . The rest 191 vectors form a matrix, which is multiplied with the output of the 3 × 3 convolution layer to generate 191 × 96 × 128 output features through reshaping. Finally, those output features are fed to the next upsampling block of M-Net. Also, the estimated µ M is added back at the end of M-Net in Fig. 2(b), which makes the remaining parts of M-Net focus on the estimation of the meanremoved depth map M − µ M U by exploiting the N-Net features ψ N .

Loss Functions
Let us describe the loss functions for training the three decoders. For G-Net, we use the ℓ 1 loss whereĜ x andĜ y are predictions of the ground-truth gradient maps G x and G y , respectively. Also, T denotes the number of valid pixels in the ground-truth. For N-Net, we use two loss terms: the ℓ 1 loss and the gradient loss. The ℓ 1 loss is defined as whereN and N are predicted and ground-truth normalized depth maps. Note that scale-invariant terms are often adopted to train monocular depth estimators [1,30,42]. However, we do not use such scale-invariant losses since normalized depth maps are already scale-invariant. Next, the gradient losses [16,33,51] for N in the horizontal direction are defined as whereN s and N s are the bilinearly scaledN and N with a scaling factor s. We compute the gradient losses at various scales, as in [33,51], by setting s to 0.5, 0.25, and 0.125. The losses L N y in the vertical direction are also used. Similarly, for M-Net, we use the loss terms L M , L M x , and L M y . In addition, we use two more loss terms. First, L µ M is defined to train the MDR block, which is given by where µ(M ) denotes the mean of depth values inM . Second, we define the logarithmic ℓ 1 loss, In this work, we adopt inverse depth representation of metric depths to match the depth order with a relative depth dataset [51]. In this case, theoretically, a metric depth can have a value in the range [0, ∞). Thus, when a metric depth is near zero, its inverted value becomes too large, which interferes with training. We overcome this problem through a simple modification. Given an original metric depth m o , its inverted metric depth m is defined as In this way, inverted metric depth values are within the range of (0, 1] and also are more evenly distributed. However, using the ℓ 1 loss L M on inverse depths has a disadvantage in learning distant depths. Suppose thatχ and χ are predicted and ground-truth metric depth values for a pixel, respectively. Then, the ℓ 1 error E is given by As χ gets larger, E becomes smaller for the same |χ − χ|. This means that the network is trained less effectively for distant regions. This problem is alleviated by employing L log M in (11).

Datasets
We use four depth datasets: one for relative depths [51] and three for metric depths [23,45,46]. When relative depth data are used in training, losses are generated from the loss terms for N-Net and G-Net only because the loss terms for M-Net cannot be computed. HR-WSI [51]: It consists of 20,378 training and 400 test images. The groundtruth disparity maps are generated by FlowNet 2.0 [19]. We use only the training data of HR-WSI. We normalize the disparity maps by (1)

Evaluation Metrics
Metric depths: We adopt the four evaluation metrics in [6], listed below. Here, M i andM i denote the ground-truth and predicted depths of pixel i, respectively. | · | denotes the number of valid pixels in a depth map. For the NYUv2 dataset, we adopt the center crop protocol [6]. Relative depths: we use two metrics for relative depths. First, WHDR (weighted human disagreement rate) [2,43,51] measures the ordinal consistency between point pairs. We follow the evaluation protocol of [51] to randomly sample 50,000 pairs in each depth map. However, WHDR is an unstable protocol, under which the performance fluctuates with each measurement. We hence use Kendall's τ [21] additionally, which considers the ordering relations of all pixel pairs. Given a ground-truth normalized depth map D and its predictionD, Kendall's τ is defined as where α(D, D) and β(D, D) are the numbers of concordant pairs and discordant pairs between D andD, respectively. Note that Kendall's τ can measure the quality of a metric depth map, as well as that of a relative one.

Implementation Details
Network architecture: We employ EfficientNet-B5 [47] as the encoder backbone. The encoder takes an 512 × 384 RGB image and generates a 16 × 12 feature with 2,048 channels. The output feature is used as the input to the three decoders. G-Net and N-Net consist of 5 upsampling blocks, each of which is composed of a bilinear interpolation layer and two 3 × 3 convolution layers with the ReLU activation. Also, in addition to the 5 upsampling blocks, M-Net includes the MDR block, located between the fourth and fifth upsampling blocks. For feature fusion in (6), ω G = ω N = 1. Training: We train the proposed algorithm in two phases. First, we train the network, after removing M-Net, for 20 epochs with an initial learning rate of 10 −4 . The learning rate is decreased by a factor of 0.1 at every fifth epoch. Second, we train the entire network, including all three decoders, jointly for 15 epochs with an initial learning rate of 10 −4 , which is decreased by a factor of 0.  at every third epoch. We use the Adam optimizer [24] with a weight decay of 10 −4 . If a relative depth is used in the second phase, losses are calculated from the loss terms for N-Net and G-Net only. Table 1 compares the proposed algorithm with conventional ones on NYUv2 dataset. Some of the conventional algorithms use only NYUv2 training data [1,3,6,7,12,15,18,25,28,30,56], while the others use extra data [41,42,49]. For fair comparisons, we train the proposed algorithm in both ways: 'Proposed' uses NYUv2 only, while 'Proposed † ' uses both HR-WSI and NYUv2. The following observations can be made from Table 1.

Performance Comparison
-'Proposed' outperforms all conventional algorithms in all metrics with no exception. For example, 'Proposed' provides a REL score of 0.100, which is 0.003 better than that of the second-best algorithm, Bhat et al. [1]. Note that both algorithms use the same encoder backbone of EfficientNet-B5 [47].  -'Proposed † ' provides the best results in five out of six metrics. For δ 2 , the proposed algorithm yields the second-best score after Ranftl et al. [42]. It is worth pointing out that Ranftl et al. uses about 20 times more training data than the proposed algorithm does. Fig. 4 compares the proposed algorithm with the conventional algorithms [1,3,6,7,30,42] qualitatively. We see that the proposed algorithm estimates the depth maps more faithfully with smaller errors. Table 2 verifies the effectiveness of the proposed algorithm on various datasets. The first two columns in Table 2 indicate the metric depth dataset and its size. 'Baseline' is a bare encoder-decoder for monocular depth estimation. Specifically, we remove G-Net and N-Net, as well as the MDR block in M-Net, from the proposed algorithm in Fig. 2 to construct 'Baseline.' For its training, only three loss terms L M , L M x , and L M y are used. 'Proposed' means the proposed algorithm without employing the 20K HR-WSI training data, while 'Proposed † ' means using the HR-WSI data additionally. The following observations can be made from Table 2.

Various Datasets
-By comparing 'Proposed' with 'Baseline,' we see that G-Net and N-Net help M-Net improve the performance of metric depth estimation by transferring edge information and relative depth information. Also, 'Proposed † ' meaningfully outperforms 'Proposed' by leveraging relative depth training data in HR-WSI, which contain no metric depth labels. -Even when only the 795 NYUv2 images are used, the proposed algorithm provides acceptable results. For example, the RMSE score of 0.417 is similar to that of the Hyunh et al.'s estimator [18] in Table 1, which uses 50K metric depth map data. In contrast, the proposed algorithm uses the 795 metric depth maps only. -The proposed algorithm also exhibits similar trends in the DIML-Indoor and SUN RGB-D datasets, which are collected using different cameras: the proposed algorithm can be trained effectively even with a small number of metric depth images. This is advantageous in practical applications in which an algorithm should be adapted for various cameras. For all datasets, 'Proposed † ' provides more accurate and more detailed depth maps, especially around chairs, tables, and desks, than 'Baseline' does.

Analysis
Ablation studies: We conduct ablation studies that add the proposed components one by one in Table 3. Here, the 17K images from NYUv2 are used for training. M, N, and G denote the three decoders. MDR * is the MDR block with µ M deactivated. † indicates the use of relative depth data. We see that all   components lead to performance improvements, especially in terms of the two relative depth metrics Kendall's τ and WHDR. Table 4 shows the effectiveness of the two-phase training scheme of the proposed algorithm. The proposed algorithm, which trains G-Net and N-Net first, shows better results than the single-phase scheme, which trains the entire network at once. Complexities and inference speeds: Table 5 compares the complexities of the proposed algorithm and the Ranftl et al.'s algorithm [42]. The proposed algorithm performs faster with a smaller number of parameters than the Ranftl et al.'s algorithm [42] does. This indicates that the performance gain of the proposed algorithm is not from the increase in complexity but from the effective use of relative depth features. Table 6 lists the complexity of each component of the proposed algorithm. The encoder spends most of the inference time, while the three decoders are relatively fast.

Conclusions
We proposed a monocular depth estimator that decomposes a metric depth map into a normalized depth map and scale features. The proposed algorithm is composed of a shared encoder with three decoders, G-Net, N-Net, and M-Net, which estimate gradient maps, a normalized depth map, and a metric depth map, respectively. G-Net features are used in N-Net, and N-Net features are used in M-Net. Moreover, we developed the MDR block for M-Net to utilize N-Net features and improve the metric depth estimation performance. Extensive experiments demonstrated that the proposed algorithm provides competitive performance and yields acceptable results even with a small metric depth dataset.