{'fraction_non_alphanumeric': 0.03395380130074008, 'fraction_numerical': 0.014039022202287508, 'mean_word_length': 4.862739942150934, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference

Monocular depth estimation is a challenging task in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks (CNNs), the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contrast to the regression based formulation. In this way, we could build upon the recent progress in dense labeling such as semantic segmentation. Second, we fuse different side-outputs from our front-end dilated convolutional neural network in a hierarchical way to exploit the multi-scale depth cues for depth estimation, which is critical to achieve scale-aware depth estimation. Third, we propose to utilize soft-weighted-sum inference instead of the hard-max inference, transforming the discretized depth score to continuous depth value. Thus, we reduce the influence of quantization error and improve the robustness of our method. Extensive experiments on the NYU Depth V2 and KITTI datasets show the superiority of our method compared with current state-of-the-art methods. Furthermore, experiments on the NYU V2 dataset reveal that our model is able to learn the probability distribution of depth.


Introduction
Depth estimation aims at predicting pixel-wise depth for a single or multiple images, which is an essential intermediate component toward 3D scene understanding. It has been shown that depth information can benefit tasks such as recognition [1,2], human computer interaction [3], and 3D model reconstruction [4]. Traditional techniques have predominantly worked with multiple images to make the problem of depth prediction well-posed, which include Nview reconstruction, structure from motion (SfM) and simultaneous localization and mapping (SLAM) [].
However depth estimation from a monocular single viewpoint lags far behind its multi-view counterpart. This is mainly due to the fact that the problem is illposed and inherently ambiguous: a single image on its own does not provide any depth cue explicitly (i.e., given a color image of a scene, there are infinite number of 3D scene structures explaining the 2D measurements exactly). When specific scene dependent knowledge is available, depth estimation or 3D reconstruction from single images can be achieved by utilizing geometric assumptions such as the "Blocks World" model [5], the "Origami World" model [6], shape from shading [7] and repetition of structures [8]. However, these cues typically work for images with specific structures and may not be applied to general scenarios.
Recently, learning based monocular depth estimation methods that predicting scene geometry directly by learning from data, have gained popularity. Typically, such approaches recast the underlying depth estimation problem in a pixel-level scene labeling pipeline by exploiting relationship between monocular image and depth. Fully-convolutional neural network has been proved to be an effective method to solve these kinds of problems. There have been considerable progress in applying deep convolutional neural network (CNN) to this problem and excellent performances have been achieved [7,8,9,10,11,12,13,14].
Albeit the above success, state-of-the-art monocular depth estimation meth-ods still fall short to handle real world challenging complex decompositions depicting multiple objects of diverse scales due to the following difficulties: 1) the serious data imbalance problem due to the perspective effect, where samples with small depths are much more than samples with large depths; 2) there are more rapid changes in depth value compared with other dense predictions tasks such as semantic labeling and 3) long range context information is needed handle the scale ambiguity in depth estimation. Even though there have been various post-processing methods to refine the estimated depth from the deep network map [7,8,9,10,11,12,13,14], the bottleneck in improving monocular depth estimation is still the specially designed CNN architecture, which is highly desired.
In this paper, we present a deep CNN based framework to tackle the above challenges, which learns the direct mapping from the color image to the corresponding depth map in an end-to-end manner. We recast monocular depth estimation as a multi-category dense labeling as contrast to the widely used regression formulation. Our network is based on the deep residual network [15], where dilated convolution and hierarchical fusion layers are designed to expand the receptive field and to fuse multi-scale depth cues. In order to reduce the influence of quantization error and improve the robustness of our method, we propose to use a soft-weighted-sum inference. Extensive experimental results show that even though we train our network as a standard classification task with the multinomial logistic loss, our network is able to learn the the probability distribution among different categories. The overall flowchart of our framework is illustrated in Fig. 1.
Our main contributions can be summarized as: • We propose a deep end-to-end learning framework to monocular depth estimation by recasting it as a classification task, where both dilated convolution and hierarchical feature fusion are used to learn the scale-aware depth cues.
• Our network is able to output the probability distribution among differ-ent depth labels. We propose a soft-weighted-sum inference, which could reduce the influence of quantization error and improve the robustness.
• Our method achieves the state-of-the-art performance on both indoor and outdoor benchmarking datasets, NYU V2 and KITTI dataset.  Residual network [15] and consists of dilated convolution and hierarchical feature fusion. Softweighted-sum inference is used to predict continuous depth values from the discrete depth labels. We also illustrate typical probability distribution of labels from the network, which shows that our classification based framework is able to learn the similarity between labels.

Related work
In this section, we briefly review related works for monocular depth estimation, which can be roughly categories as conventional MRF/CRF based methods and deep learning based methods.
MRF/CRF Based Methods: Seminal work by Saxena et al. [16,17] tackles the problem with a multi-scale Markov Random Field (MRF) model, with the parameters of the model learned through supervised learning. Liu et al. [18] estimated the depth map from predicted semantic labels, achieving improved 4 performance with a simpler MRF model. Ladicky et al. [19] showed that perspective geometry can be used to improve results and demonstrated how scene labeling and depth estimation can benefit each other under a unified framework, where a pixel-wise classifier was proposed to jointly predict a semantic class and a depth label from a single image. Besides these parametric methods, other works such as [20,21,22] recast monocular depth estimation in a nonparametric fashion, where the whole depth map is inferred from candidate depth maps. Liu et al. [21] proposed a discrete-continue CRFs, which aims to avoid jointly. Wang et al. [9] proposed a CNN architecture for joint semantic labeling and monocular depth prediction. Chen et al. [24] proposed an algorithm to estimate metric depth using annotations of relative depth.
Very recently, Laina et al. [12] proposed using the Huber loss instead of the L 2 loss to deal with the long tail effect of the depth distribution. Cao et al. [11] demonstrated that formulating depth estimation as a classification task could achieve better results than regression with L 2 loss, while insufficient analysis is given for the success. In addition, different with our method, they used hard-max inference in the testing phase. Xu et al. [13] proposed a Multi-Scale 5 Continuous CRFs to better extract the hierarchical information and improve the smoothness of the final results. Our hierarchical information fusion strategy is much simpler than [13], while we also achieve comparable results.
Unsupervised monocular depth learning Besides the above methods using ground truth depth maps to supervise the network learning, there is another group of methods that using novel view synthesis to supervised the network learning by exploiting the availability of stereo images and image sequences [25] [14] [26] [27] citeUnsupervised-Depth-Motion. Garg et al. [25] proposed to train a network for monocular depth estimation using an image reconstruction loss, where a Taylor approximation is performed to linearize the loss. Godard et al. [14] replaced the use of explicit depth data during training with easier-to-obtain binocular stereo footage, which enforces consistency between the disparities produced relative to both the left and right images, leading to improved performance and robustness compared to existing approaches. Along this pipeline, Zhou et al. [26] presented an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences based on image warping to evaluate the image error. Kuznietsov et al. [27] learned depth in a semi-supervised way, where sparse ground-truth depth and photoconsistency are jointly used. Ummenhofer et al. [28] trained a convolutional network end-to-end to compute depth and camera motion from successive, unconstrained image pairs, where the architecture is composed of multiple stacked encoder-decoder networks.
The key supervision signal for these "unsupervised" methods comes from the task of novel view synthesis: given one input view of a scene, synthesize a new image of the scene seen from a different camera pose. Essentially, pairs of rectified stereo images or consecutive image frames have already encode the depth information implicitly.
Our work is also related to the works on FCN (fully convolutional network) based dense labeling. Long et al. [29] proposed the fully convolution neural network for semantic segmentation, which is widely used in other dense labeling problems. Hariharan et al. [30] presented that low-level CNN feature is bet-6 ter to the boundary preserving and object location. Recently, Yu et al. [31] demonstrated that dilated convolution could expand the receptive field of the corresponding neuron while keeping the resolution of the feature map. Chen [32] successfully utilized the dilated convolution on the semantic problem and show how to build them on the pre-trained CNN.

Our Framework
Targeting at handling the real world challenges with the current state-of-theart methods, we propose a deep end-to-end learning framework to monocular depth estimation, which learns the direct mapping from a color image to the corresponding depth map. Our framework to monocular depth estimation consists of two stages: model training with classification loss and inference with softweighted sum. First, by recasting monocular depth estimation as multi-class labeling, we design an hierarchical fusion dilated CNN to learn the mapping from an RGB image to the corresponding depth score map directly. Our network architecture hierarchically fuses multi-scale depth features, which is important to achieve scale-aware monocular depth estimation. Second, we propose a softweighted-sum inference as contrast to the hard-max inference, which transfers the discretized depth scores to continuous depth values. In this way, we could reduce the influence of quantization error and improve the robustness.

Network Architecture
Our CNN architecture is illustrated in Fig. 2, in which the weights are initialized from a pre-trained 152 layers deep residual CNN (ResNet) [15]. Different from existing deep network [33], ResNet [15] explicitly learns residual functions with reference to the layer inputs, which makes it easier to optimize with higher accuracy from considerably increased network depth. ResNet [15] was originally designed for image classification. In this work, we re-purpose it to make it suitable to our depth estimation task by • Removing all the fully connect layers. In this way, we greatly reduce the number of model parameters as most of the parameters are in the fully connect layers [10]. Although preserving the fully connect layers is beneficial to extract long range context information, our experiments show that it is unnecessary in our network thanks to dilated convolution.
• Taking advantage of the dilated convolution [31]. Dilated convolution could expand the receptive field of the neuron without increasing the number of model parameters. Furthermore, with the dilated convolution, we could remove some pooling layers without decreasing the size of receptive field of correspondent neurons. In addition, we could keep the resolution of the feature map and final results, i.e., the output resolution has been increased.  Figure 2: Illustration of our network architecture. The detail of the basic residual block could be referred to [15]. ×n means the block repeats n times. We present all the hyperparameters of convolution and pooling layers. All the convolution layers are followed by batch normalization layer except for the last one. /2 means the layer's stride is 2. * 4 means the deconv layer's stride is 4. Dilation shows the dilated ratio of the corresponding parts.

Dilated Convolution:
Recently, dilated convolution [31] has been successfully utilized in deep convolutional neural network, which could expands the field of perception without increasing the number of model parameters. Spe-8 cially, let F : Z 2 → R be a discrete function. Let Ω r = [r, r] 2 ∩ Z 2 and let k : Ω r → R be a discrete filter of size (2r + 1) 2 . The discrete convolution filter * can be expressed as We now generalize this operator. Let l be a dilation factor and let * l be defined We refer to * l as a dilated convolution or an l-dilated convolution. The conventional discrete convolution * is simply the 1-dilated convolution. An illustration of dilated convolution could be found in Fig. 3. In conclusion, we briefly summarize our final network design. Typically, the pre-trained residual network is consisted of 4 parts. We remove the max-pooling layer in the last 2 parts and expand the corresponding convolution kernel with dilation 2 and 4 respectively. Then, a concatenation layer is added to fuse the hierarchical multi-scale informations from layers L1 − L4. The last two layers of our network are convolution layer and deconvolution layer. The parameters setting is presented in Fig. 2.

Soft-Weighted-Sum Inference
We reformulate depth estimation as classification task by equally discretizing the depth value in log space. Specifically, where l is the quantized label, d is the continuous depth value, d min is the minimum depth value in the dataset or set to be a small value like 0.1. q is the width of the quantization bin.
With the quantization label, we train our network with the multinomial logistic loss.
where N is the number of training samples, exp(θ (k)T x (i) ) is the probability of label k of sample i, and k is the ground truth label.
In the testing stage, we propose to use the soft-weighted-sum inference. It is worth noting that, this method transforms the predicted score to the continuous depth value in a natural way. Specifically: where w is the weight vector of depth bins. p is the output score. In our experiments, we set the number of bins to 200.

Data Augmentation
Although the training dataset is of ten thousands images, we still find the data augmentation is important to improve the final performance. In this work, we augment our dataset by 4 times for both the NYU v2 and the KITTI dataset.
The augmentation methods we utilized include: • Color: Color channels are multiplied by a factor c ∈ [0.9, 1.1] randomly.
• Scale: We scale the input image by a factor of s ∈ [1.3, 1.5] randomly and crop the center patch of images to match the network input size.
• Left-Right flips: We flip left and right images horizontally.
• Rotation: We rotate the input image randomly by a factor of r ∈ [−5, 5].

Implementation details
Before proceeding to the experimental results, we give implementation details of our method. Our implementation is based on the efficient CNN toolbox: caffe [34] with an NVIDIA Tesla Titian X GPU.
The proposed network is trained by using stochastic gradient decent with batch size of 1 (This size is too small, thus we average the gradient of 8 iterations for one back-propagation), momentum of 0.9, and weight decay of 0.0004.
Weights are initialized by the pre-trained model from ResNet [15]. The network is trained with iterations of 50k by a fixed learning rate 0.001 in the first 30k iterations, then divided by 10 every 10k iterations.

Experimental Results
In this section, we report our experimental results on monocular depth estimation for both outdoor and indoor scenes. We used the NYU V2 dataset, and the KITTI dataset, as they are the the largest open dataset we can access at present. We compared our method with the state-of-the-art methods published recently.
For quantitative evaluation, we report errors obtained with the following metrics, which have been extensively used in [16,18,23,19,21].
where d is the ground truth depth,d is the estimated depth, and T denotes the set of all points in the images.

NYU V2 dataset
The NYU V2 dataset [4] contains around 240k RGB-depth image pairs, of which comes from 464 scenes, captured with a Microsoft Kinect. The official split consists of 249 training and 215 testing scenes. We equally sampled frames out of each training sequence, resulting in approximately 24k unique images.
After off-line augmentations, our dataset comprises of approximately 96k RGB-D image pairs. We fill in the invalid pixels of the raw depth map with the "colorization" method, which is provided in the toolbox of NYU V2 dataset [4].
The original image resolution is 480 × 640. We downsampled the images to 240 × 320 as our network input. The resolution of the our network output is 120 × 160, which is half of the input size. In this dataset, we quantize the depth value into 200 bins.
In Fig. 4, we provide a qualitative comparison of our method with [12] and [23]. (We compare with these methods due to they published their results and they are the state-of-the-art methods at present). From Fig.4, it is clear to observe that our results are of high visual quality, although we have not applied any post-processing methods.

KITTI dataset
The KITTI dataset [35] consists of a large number of outdoor street scene images of the resolution 376 × 1242. We utilized the "Eigen" training/testing split, which consists of 22600 training images and 697 testing images. We fill in the invalid pixel of the raw depth map with the "colorization" method, which is provided in the toolbox of NYU V2 dataset [4]. Garg et al. [25] and Godard et al. [14].
The original image resolution is 376 × 1240. We downsampled the images to 188 × 620 as our network input. The resolution of the our network output is 94 × 310, which is half of the input size. For this dataset, we quantize the depth value into 50 bins.
In Table 2, we compared our method with the recent published state-of-theart methods [23,25,14].
In Fig. 5, we provide a qualitative comparison of our method with [10] and [14]. (We compare with these methods due to they published their results and they are the state-of-the-art methods at present). From  [12]. (e) Results of [10]

Performance Analysis
In this section, we present more analysis of our model, where the experiments are conducted on the NYU V2 dataset. First, we present a component analysis of our network architecture design, i.e., the contribution of each component.
Second, we analyze the distribution of our network output, which demonstrates the necessary of our soft-weighted-sum inference strategy.

Effect of Soft-Weighted-Sum Inference
One important contribution of this work is the proposed soft-weighted-sum inference. Here, we would like to elaborate the necessity and effectiveness of it.
Firstly, we give the probability distribution variation of randomly selected positions along the training in Fig. 6. The most interesting thing is that: In the training phase, we utilize the multinomial logistic loss, which means we don't specially discriminate the distance between the "nearby" and "further" classes.
While, the probability distribution is rather clustered. More interestingly, the probability distribution roughly follow the Gaussian distribution, which means it is symmetric. At last, as the training goes on, the distribution of probability is becoming more concentrated, but always maintains symmetry similar to that of Gaussian distribution.
Secondly, we use the hard-max inference and give the confusion matrix in Thirdly, we increase the number of depth bins. Under the same training setting, we present the variation of "pixel accuracy" and the relative errors in Tab. 4. With the increase of number of bins, the "pixel accuracy" drop dramatically, while the relative error keeps stable. This trend presents that: 1) At present, the network cannot distinguish the very detailed distance variation even we train it with the very detailed "label". In other words, "depth perception" ability of the network is limited.

Conclusions
In this paper, we have proposed a deep end-to-end classification based framework to monocular depth estimation. By using both dilated convolution and hierarchical fusion of multi-scale features, our framework is able to deal with the real world difficulties in multi-scale depth estimation. Extensive experiments on both indoor and outdoor benchmarking datasets show the superiority of our method compared with the current state-of-the-art methods. More importantly, experiments also demonstrate that our model is able to learn a probability distribution among different depth labels, which inspires the proposed soft-weighted-sum inference.