{'fraction_non_alphanumeric': 0.03073943068890012, 'fraction_numerical': 0.009727134916624558, 'mean_word_length': 4.68293850203398, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A New Distributional Ranking Loss With Uncertainty: Illustrated in Relative Depth Estimation

We propose a new approach for the problem of relative depth estimation from a single image. Instead of directly regressing over depth scores, we formulate the problem as estimation of a probability distribution over depth and aim to learn the parameters of the distributions which maximize the likelihood of the given data. To train our model, we propose a new ranking loss, Distributional Loss, which tries to increase the probability of farther pixel's depth being greater than the closer pixel's depth. Our proposed approach allows our model to output confidence in its estimation in the form of standard deviation of the distribution. We achieve state of the art results against a number of baselines while providing confidence in our estimations. Our analysis show that estimated confidence is actually a good indicator of accuracy. We investigate the usage of confidence information in a downstream task of metric depth estimation, to increase its performance.


Introduction
Depth is a key factor of a scene and it has always been an important challenge to estimate it, especially from monocular images. With the advancements in the deep learning techniques, we started to see very successful attempts in monocular depth estimation task such as [9,14,10], where the aim is to estimate absolute depth. However, most of the state of the art works focusing on absolute depth estimation, utilize limited datasets such as indoor only (e.g. NYUDv2) or outdoor only (e.g. KITTI) datasets. While models trained on limited datasets perform well on their immediate training domain, they do not generalize well to images coming from different distributions.
In order to be able to estimate depth in-the-wild, a reformulation of the depth estimation problem is employed, namely relative depth estimation. With this reformulation, diverse datasets are collected and models that work in-thewild are trained [6,36,7]. However, these approaches are  Fig. 1a shows the input RGB image where we see letters on the side of a vehicle. However in Fig.  1b, it can be seen that the model fails to account for the smooth depth change across the letters, and produces unreliable depth estimates that normally should overlook the letter boundaries. While this mistake would go unnoticed for previous approaches, our model learns to express its estimation confidence. Fig. 1c clearly shows that the model is not confident in its estimations for the letters, indicated by darker pixels.
not capable of expressing model confidence, which we believe to be a very important information for the depth estimation problem, as the estimated depth map is usually utilized for subsequent tasks or decision making processes. For instance, both [23,31] utilize uncertainty maps in 3D reconstruction to increase performance and robustness. While previous approaches directly estimate depth scores as a regression task, we treat depth as it is normally distributed, parameterized by mean µ and standard deviation σ, and regress over these parameters for each pixel. We believe that this representation is more natural as the ground truth information is also uncertain about the actual depth value. Furthermore, this representation effectively makes our model capable of displaying its confidence in terms of σ. In order to learn from ordinal relations of pixels, we propose a novel loss function, a distributional loss, which attempts to increase the probability of farther pixel's depth being greater than closer pixel's depth. An illustrative ex-ample is shown in Figure 1, where our model outputs a depth map as well as a separate confidence map for the estimated depth, which points to regions of uncertainty in the estimation.
The contributions of our work are as follows: • We formulate the problem of depth estimation as estimating a probability distribution over depth where 1 σ can be considered as the confidence. Given the ground truth information, we believe this formulation is more intuitive and it allows us to output confidence.
• We devise a new ranking loss, the distributional loss, that allows us to learn parameters of the distribution for each pixel from ordinal relations of pixels.
• We evaluate our approach against a number of baselines in the literature and achieve state of the art performances.
• We analyze the confidence output and empirically exhibit its usefulness.

Background
Absolute depth estimation Early works in the field utilized hand crafted features and Markov Random Fields while incorporating human expertise in terms of hand designed constraints on optimization process [13,19,29,30,16]. With the increasing success of convolutional neural networks on vision problems, a number of works employed convolutional neural networks in a standard supervised learning setting [9,14,3,10]. Additional complementary tasks were also utilized to increase the performance [8,34,24,38,39,40,5]. In order to eliminate the need for real world ground truth data, number of works do selfsupervised learning [11,12,41,33,26], while [27] used synthetic images.
Relative depth estimation To the best of our knowledge, Zoran et al. [42] did the first attempt at relative depth estimation by classifying ordinal relations of pixel pairs. Since it is infeasible to classify all possible pixel pairs, they superpixelated the input image and only compared centers of superpixels, assuming that superpixels represent homogeneous depth patches. [6,7] estimated a dense score map in a regression setting and used a pairwise ranking loss to learn from ordinal relations. In the same framework, [36] applied an improved pairwise ranking loss which focuses on a set of hard pairs and [37] proposed a sampling strategy that focuses on image and object edges. [18] employed a listwise ranking loss which allowed their model to focus more on closer pixels.
In-the-wild datasets First dataset with relative depth annotations is Depth in the Wild (DIW) [6]. It consists of randomly sampled images from internet and ground truth ordinal relation of one pair of pixels per image, annotated by human annotators, and has an official train test split. Afterwards, two other datasets with relative depth annotations are proposed: YouTube3D [7] and Relative Depth from Web (RedWeb) [36]. While YouTube3D offers sparse ground truth information, RedWeb dataset has dense relative depth annotation that can be acquired from given ground truth score map. YouTube3D and RedWeb do not have an official train test split. Works that use these datasets, use the whole dataset for training and test their performance on DIW test split.
Ranking Ranking methods can be divided into two main categories based on whether they directly optimize ranking measures. [2,28] proposes differentiable approximations for ranking measures which allow them to be used in the optimization process. On the other hand, a number of works optimize surrogate measures in a pairwise [1,32] or a listwise manner [4,25,35,15]. Yet none of them learns to output confidence.

Approach
In relative depth estimation, the ground truth information consists of pixels' ordinal relations which falls in three categories as where Ω represents the set of all pixels, d i is the metric depth of pixel i, and r ij represents the ordinal relations of pixel i and j. The aim is to predict the ordinal relation for pixel pair i and j,r ij , to minimize the following objective: where 1 () is the indicator function that evaluates to 1 if r ij =r ij is true, otherwise it evaluates to 0. We explain our approach to solve relative depth estimation problem in two main parts. First, we discuss our formulation of the relative depth estimation problem. Next, we present the ranking loss that works with the proposed formulation. Figure 2 depicts the overall framework that we use to train neural networks for relative depth estimation problem. Note that our approach does not depend on any particular neural network architecture, or a particular model for that matter. Any parameterized, differentiable model can be used as a ranking function. Additionally, we propose a general approach, in a sense that it is applicable to any other ranking problems. Yet throughout the paper, we are going to discuss our approach particularly for the relative depth estimation problem as we showcase our approach in this domain.

Formulation
As previous works [6,36,7], we follow a pairwise approach. We consider two relations between pixels where either pixel i is farther compared to pixel j, or pixel j is farther compared to pixel i. We omit the case where both pixels are at the same depth as DIW and YouTube3D datasets do not have examples of equality case.
To solve the relative depth estimation problem, previous works try to estimate a score for each pixel that can be used as ordinal relations as follows: wheres i is the estimated score for pixel i. However, we neither know the ground truth values for s i , ∀i, nor do care about actual value ofs i , ∀i, as long as they satisfy Equation (3). In these circumstances, we believe it is more intuitive to estimate a probability distribution over depth values, rather than scalar values. In this work, we investigate the employment of one natural choice for this task, that is the normal distribution.
Probability of depth of pixel i conditional on an image I, is modelled as a normal distribution whose mean µ i and standard deviation σ i is estimated from the image I by a neural network model g(I; θ): We formulate the relative depth estimation problem as a maximum likelihood estimation problem and try to find the optimal parametersθ that maximizes the probability of observed data y as follows: where Θ represents the parameter space, and h is the data likelihood function, which is to-be-defined (Eq. 6).
With this formulation, we can consider σ i as an uncertainty information for that pixel i's depth estimation. As σ i increases, the range of values, into which the pixel i's score value falls, increases. In practice, we model the function g(I; θ) with a neural network model. When we first initialize the network, we observe small values in the outputs, indicating that there is very small uncertainty in model's estimations. This is counter intuitive, since at the start of the training we expect the uncertainty to start at high levels and decrease as the training proceeds. Also, estimating σ i is not numerically stable for the very same reason. To overcome this problem, we propose to learn to estimate the reciprocal of σ i , i.e. 1 σi , which can be interpreted as the "confidence". This formulation reflects the model's knowledge in a better way, particularly addressing the low confidence at the start of the training. We also experiment with both of the formulations and empirically show that learning the reciprocal, i.e. the confidence rather than the uncertainty, is better.

Distributional Loss (DL)
In this section, we introduce our proposed ranking loss, the distributional loss. It works in a pairwise fashion. Since ground truth information consists of ordinal relations of pixels, we calculate the likelihood of the observed training data as follows: where P refers to the probability of depth of pixel i being greater than that of pixel j or vice versa depending on its arguments, and 1(·) refers to the indicator function.
To simplify things, let us assume where f refers to the pixel that is supposed to be farther and c refers to the pixel that is supposed to be closer for a particular pairwise relation. Equation (6) now becomes which can be maximized by minimizing the following distributional loss By rearranging the terms of P (d f > d c ), we get which can be calculated as where Φ is the error function erf(·) in mathematics that has differentiable implementations in popular libraries. Overall, our distributional loss in its full form is as follows: In test time, we chooser ij as follows: In practice, we only compare µ i and µ j as P (d i > d j ) > 0.5 only when µ i > µ j or vice versa.
Behaviour of the DL loss function Let us examine how the proposed DL loss function behaves under certain conditions. First we fix the standard deviation σ 2 f + σ 2 c to 1 and plot the change in the loss as µ f − µ c changes. As it can be seen from Figure 3, the DL loss decreases monotonously as the µ f − µ c increases. Similar to the ranking losses used in previous works [6,36,7], the distributional loss also encourages bigger differences between scores, but only to some extent. The loss vanishes as µ z increases since as long as P (d f > d c ) > 0.5 is satisfied, we gain no extra benefit in terms of our objective Equation (2).
To see the effect of model's uncertainty predictions on the DL loss, we fix the µ f − µ c to 1 and −1, where the model's estimation is correct and incorrect, respectively. Figure 4a shows that when the model's estimation is correct, increasing the uncertainty increases the loss. Similarly, Figure 4b shows that the DL loss decreases as the uncertainty increases when the model's estimation is incorrect. To sum up, our loss encourages confidence when the estimation is correct, encourages uncertainty when the estimation is incorrect and to decrease the loss, the model would learn to predict the correct confidence in its estimations.
Derivatives of the DL loss function First, we examine the derivatives of our loss function in Equation (12). Its derivatives with respect to µ f and µ c are given in Equation (14). Note that they only differ in sign, means that they always change in opposite direction. Therefore, we only plot and discuss gradients of µ f .  Figure 5 shows the gradients of µ f by fixing the σ z and plotting it against µ z (Fig. 5a), and by fixing the µ z to +1 and −1 and plotting it against σ z (Fig. 5b and 5c, respectively). Since µ f is the estimated mean for the pixel that is supposed to be farther away, it receives only negative gradients which increases its value. Figure 5a shows that the absolute value of the gradients increases as the mistake, |µ z | when µ z < 0, gets bigger, and it vanishes when the estimation is corrected, µ z > 0, reflecting the behaviour of the loss function shown in Figure 3. When we fix the µ z to 1 where the model's prediction is correct (Fig. 5b), the absolute value of gradients which µ f receives increases with the uncertainty to some extent meaning that it increases µ f more and more aggressively, which is intuitively as expected. Then the rate of the gradient decreases and levels off while the uncertainty keeps increasing. This behaviour is open to interpretation. As the confidence decreases, this loss updates the mean score less aggressively than its value at starting points. When the model's prediction is wrong (Fig. 5c), µ f changes rapidly if the model is confident and the speed of change decreases as the confidence decreases. Again, this reflects the behaviour of the DL loss that can be seen in Figure 4b.
Equation (15) shows the derivatives with respect to σ f and σ c , which are equal except ∂DL ∂σ f scales with σ f and ∂DL ∂σc scales with σ c . Note that the sign of the gradients depend on µ z . When the model's prediction is correct, µ z > 0, both σ f and σ c receives positive gradients and confidence (a) σz = 1 increases or vice versa. Therefore, we only plot and discuss the gradients of σ f .
(15) Figure 6 shows the gradients of σ f by fixing the σ z and plotting it against µ z (Fig. 6a), and by fixing the µ z to +1 and −1 and plotting it against σ c (Fig. 6b and 6c, respectively). As it can be seen in Figure 6a, when the model's prediction is incorrect, µ z < 0, increase in the uncertainty increases as the mistake, |µ z |, gets bigger. However, when the model's prediction is correct, µ z > 0, the decrease in the uncertainty vanishes as µ z increases since as long as P (d f > d c ) > 0.5, increase in the µ z is only rewarded to some extend by the loss function (see Fig. 3). Furthermore, in both plots of (b) and (c), ∂DL ∂σ f vanishes as σ c increases, implying that the information gain decreases as the uncertainty increases.

Experiments and Results
We divide our experiments into three categories. First, we conduct an analysis where we examine the confidence prediction on the DIW test split, and devise an experiment to empirically show the usefulness of confidence prediction for the metric depth estimation task. Next, we compare learning standard deviation σ directly and reciprocal of it 1 σ . Lastly, we compare our proposed approach with state of the art baselines to show our method's performance for the relative depth estimation task.
For relative depth estimation tasks, we report weighted human disagreement rate (WHDR) [42] as where ω ij is the human confidence weight and set to 1 for DIW test split, r ij andr ij represent ground truth and predicted ordinal relations, respectively. For metric depth estimation tasks, we report the metrics from [9].
We measure calibration performance, which is the degree of consistency between model's predicted probabilities of outcomes and the true probabilities of those outcomes. To this end, we use expected calibration error (ECE) [21], its variants maximum calibration error (MCE) [21] and adaptive ECE (AdaECE) [20], and reliability plots [22]. They are defined for classification settings where the model's probability output (interpreted as confidence) for a • reliability plots: plot of accuracies at each bin as a bar chart.
All the experiments are conducted with the same setting. We perform on-the-fly data augmentation. Specifically, we horizontally flip the image, rescale and crop it to the input size of 384×384, and apply rotation. We experiment with a common architecture used in previous works: EncDecRes-Net [36]. We use stochastic gradient descent (SGD) with cosine annealing learning rate scheduler [17] which cyclically changes learning rate between [1e − 3, 1e − 7] and completes a cycle at every 5 epoch. Batch size is 8 in all experiments. All of the hyperparameters are chosen heuristically due to limited computational resources.

Investigating confidence prediction
To demonstrate the effectiveness of our approach for confidence prediction, we measure the model calibration of EncDecResNet trained with DL loss on RedWeb training set, which we refer to as DL EDR. We choose r ij = 1 as positive class and calculate the class probability P (r ij = 1) as P (d i > d j ), as given in Equation (11). However, this probability is affected by the µ i and µ j predictions as well. To better show the effect of confidence prediction, we also use µ i −µ j as P (r ij = 1) by mapping it to [0, 1] range (represented as P (r ij = 1) = µ i − µ j ). The latter formulation essentially calculates the confidence based on the distances between the estimated centers of both pixels' depth distribution, i.e. confidence increases as the distances between the centers increases.   Table 1 shows the calibration measures ECE [21], AdaECE [20], and MCE [21], and Figure 7 shows the reliability plots [22] of DL EDR with different interpretations of P (r ij = 1). Both calibration measures and reliability plots indicate that µ i − µ j alone, is not a good indicator of confidence. However, when we utilize confidence predictions as well, model output becomes almost perfectly calibrated, indicating that confidence predictions reflects the model's expected accuracy.
To empirically show the usefulness of confidence information, we conduct the following experiment. We train the EncDecResNet of [36] on the NYUDv2 dataset for the absolute depth estimation task. First, we train the network using RGB images and the corresponding score map (SM) estimation from DL EDR as an input (RGB+SM, abbreviated as W/O CM). Next, we also input the confidence map (CM) prediction of DL EDR (RGB+SM+CM, abbreviated as W/ CM) and repeat the training. Table 2 shows the results. Inputting confidence map alongside other inputs increases the performance in most of the metrics, especially the most challenging accuracy, thr = 1.25. We conjecture that confidence predictions allow network to employ different strategies for parts that are likely to have wrong relative depth score, hence the performance gain. We also conduct experiments to empirically show the effectiveness of learning reciprocal of the standard deviation, 1 σ . We train EncDecResNet on RedWeb dataset and experiment with learning standard deviation. In this case, we can interpret the second channel of the output as uncertainty map. We also repeat the same experiment with treating networks second output as 1 σ in loss formulation. In this case, we can interpret the second channel of the output as confidence map. Table 3 shows the results which indicate that learning confidence performs much better when compared to learning uncertainty. In the uncertainty version of the model, we observe that standard deviation output of the model diverges very quickly and the model converges to a local optima.

Comparison with state of the art
To show the effectiveness of our approach for the relative depth estimation task, we compare our results with several similar works [6,36,7]. We use the following naming convention in method names for ease of understanding. The first part indicates the original work that publishes the given result. The second part represents the neural network model, HG being Hourglass [6] and EDR being EncDe-cResNet [36]. To train EDR model with the proposed approach, we only add an additional head with Leaky ReLU activation for confidence output without any further modification. The last part represents the loss that is used where R is the ranking loss in [6], IR is the improved ranking loss in [36], and DL is the proposed distributional loss. Following previous works [36,7], we use ImageNet pretrained weights for initializing encoder part of the EDR in all experiments. Table 4 shows results on DIW test split. We achieve state of the art performances on all experimented datasets. We believe our good performances in all datasets with heuristically chosen parameters indicate that our method is not sensitive to hyperparameters. Figure 8 shows qualitative examples from DIW test split. We observe that model trained on RedWeb dataset produce sharper results which indicates that number of annotated pixel pairs is important for visually better results since RedWeb dataset has dense annotation. In Figure 9, we see that model has indeed low confidence for parts where the depth estimation is not accurate or for parts where the depth estimation is hard to make such as background or edges. For instance, in the first row of images, although the wall background seems parallel to the camera plane, the estimated depth is larger on the right side of the person than that of the left. This is marked by a lower confidence estimate. Similarly, see the indicated regions (by dashed green boxes) where the confidence map correctly reproduces the expected lower confidence scores.

Conclusion
In this paper, we propose a new pairwise ranking approach, and illustrate it in the problem of relative depth estimation. We estimate the probability distribution over depth values, and maximize the likelihood of the observed pairwise ordinal relations by training a neural network model with the proposed DL loss. With this formulation, we achieve better or comparable performances with prior art while outputting confidence for estimations as well. The new ranking distributional loss with uncertainty that is presented in this work is not specific to relative depth estimation, and can be utilized in other problems involving ordinal relations between measurements.