{'fraction_non_alphanumeric': 0.03285220933321527, 'fraction_numerical': 0.013548215708846187, 'mean_word_length': 5.0331196581196584, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Deterministic Uncertainty for Semantic Segmentation

We extend Deep Deterministic Uncertainty (DDU), a method for uncertainty estimation using feature space densities, to semantic segmentation. DDU enables quantifying and disentangling epistemic and aleatoric uncertainty in a single forward pass through the model. We study the similarity of feature representations of pixels at different locations for the same class and conclude that it is feasible to apply DDU location independently, which leads to a significant reduction in memory consumption compared to pixel dependent DDU. Using the DeepLab-v3+ architecture on Pascal VOC 2012, we show that DDU improves upon MC Dropout and Deep Ensembles while being significantly faster to compute.


Introduction
With the increasing deployment of deep learning models in safety critical applications like autonomous driving (Huang & Chen, 2020) and medical diagnosis (Esteva et al., 2017), it is imperative for such models to be able to quantify their uncertainty reliably, in addition to making accurate predictions. A significant amount of research has been conducted in this direction and several methods have been introduced in the context of classification (Gal & Ghahramani, 2016;Lakshminarayanan et al., 2017;Blundell et al., 2015). These methods require several forward passes through the model rendering such methods practically infeasible for adoption in large scale applications like semantic segmentation (Long et al., 2015), where dense pixel-wise predictions are necessary, often in real time.
DUQ (Van Amersfoort et al., 2020) and SNGP (Liu et al., 2020) propose using feature extractors with certain inductive biases to impose a bi-Lipschitz constraint on the feature space. They then use a distance aware layer, either an RBF or a Gaussian Process trained end-to-end with the feature extractor. However, both these methods require extensive changes to the model architecture and training setup, with additional hyperparameters which need to be fine-tuned. DDU (Mukhoti et al., 2021) shows that using feature space density with proper inductive biases can capture uncertainty and avoids the problem of feature collapse (Van Amersfoort et al., 2020). Due to feature collapse, Out-of-distribution (OoD) samples are often mapped to in-distribution regions in the feature space, making the model overconfident on such inputs. Hence, in order to capture uncertainty through feature space density, one needs to use proper inductive biases on the model architecture.
There are two kinds of uncertainty which are important in deep learning literature: epistemic uncertainty, which captures what the model does not know, is high for unseen or OoD inputs and can be reduced with more training data and, aleatoric uncertainty, which captures ambiguity and observation noise in in-distribution samples (Kendall & Gal, 2017). In DDU, the epistemic uncertainty is quantified using a feature space density, while the entropy of the softmax distribution can be used to estimate aleatoric uncertainty.
In this paper, we apply and extend DDU to the task of semantic segmentation (Long et al., 2015), where each pixel of a given input image is classified to produce an output which has the same spatial dimensions as the input. We choose semantic segmentation in particular as it forms an excellent example of an application with class imbalance and therefore, requires reliable epistemic uncertainty estimates. Furthermore, state-of-the-art models for semantic segmentation (Chen et al., 2017;Zhao et al., 2017;Wang et al., 2020) are large and conventional uncertainty quantification methods like MC Dropout (Gal & Ghahramani, 2016) and Deep Ensembles (Lakshminarayanan et al., 2017) are often prohibitively expensive on such models (Kendall et al., 2015;Mukhoti & Gal, 2018).
The paper is organised as follows: in Section 2, we describe how DDU can be extended for semantic segmentation and in Section 3, we provide results using the well-known

DDU in Semantic Segmentation
In this section, we provide details on how DDU can be extended to obtain epistemic and aleatoric uncertainty estimates in semantic segmentation.
A brief introduction to DDU: As described in Mukhoti et al. (2021) in the context of multiclass classification, after training a model with a bi-Lipschitz constraint, we can compute the feature space means and covariances per class using a single pass over all the training samples. The feature space means and covariance matrices can then be used to fit a Gaussian Discriminant Analysis (GDA) (Murphy, 2012). Let z be the feature representation for a given input x, i.e., z = f θ (x) where θ represents model parameters. Then the feature density p(z) is computed by marginalizing the density over all classes as where p(z|c) is obtained from the GDA and p(c) can be computed directly from the training set. The feature density thus computed can be used to estimate model confidence (opposite of epistemic uncertainty). At the same time, for indistribution samples, the entropy of the softmax distribution H[p(y|x, ω)] can be used to capture aleatoric uncertainty.
Pixel-independent class-wise means and covariances: In semantic segmentation, each pixel has a prediction attached to it and a corresponding softmax distribution. A natural question to ask then is whether to compute means and covariance matrices per pixel in order to fit a GDA to the semantic segmentation model. Fortunately, we find that such is not the case and we can indeed compute means and covariance matrices independent of pixels just like in multi-class classification (thereby enforcing invariance). To see this, in Figure 2, we plot the L2 distances between the feature space means of all pairs of classes in the Pascal VOC validation set for two "distant" pixels. We find that the means of the same class are much closer together as compared to other classes irrespective of where the pixels are located. This makes intuitive sense as the convolution kernel (a linear operator) which converts the feature space representations into logits is shared across the entire feature space representation.
Computing feature density: Following the rationale above, we fit a GDA assuming pixels to be independent samples. Hence, we obtain one mean and one covariance matrix per class (not per pixel) and can apply Equation (1) to obtain the feature density per pixel, given an input image. Separately, we can also obtain the per pixel softmax entropy from the model. Using these two we can disentangle aleatoric and epistemic uncertainty with a single deterministic model in semantic segmentation. We present a schematic diagram of this process in Figure 1.

Experiments
In this section, we evaluate DDU on semantic segmentation using the well-known Pascal VOC (Everingham et al., 2010) dataset and compare it with three other uncertainty baselines widely applied in practice: softmax entropy, MC Dropout (Gal & Ghahramani, 2016) and Deep Ensembles (Lakshminarayanan et al., 2017).  (Hariharan et al., 2011) using SGD as the optimiser with a momentum of 0.9 and a weight decay of 5e − 4. We set the initial learning rate to 0.007 with a polynomial decay during the course of training. Finally, we trained using a batch size of 32 parallelized over 4 GPUs.
Baselines & Uncertainty metrics: As mentioned before, we compare DDU with 3 well-known baselines: 1. Softmax Entropy, one of the most commonly used metrics for uncertainty is the entropy H[p(y|x, ω)] of the softmax distribution p(y|x, ω) (Hendrycks & Gimpel, 2016). This metric is often preferred due to its simplicity and lack of computational overhead. Softmax entropy is known to capture aleatoric uncertainty for in-distribution samples (Mukhoti et al., 2021). However, it cannot capture epistemic uncertainty reliably (eg. for OoD inputs). Metrics for evaluation: In order to evaluate the quality of uncertainty in semantic segmentation, we use the metrics proposed in (Mukhoti & Gal, 2018): p(accurate-certain), p(uncertain-inaccurate) and PAVPU. The metric p(accurate-certain) measures the probability of a prediction being accurate given that the model is confident on the prediction. Similarly, p(uncertain-inaccurate) measures the probability of the model being uncertain on inaccurate predictions. PAVPU computes the probability of the model being confident on an accurate prediction or uncertain on an inaccurate one. A good model should ideally have high values on all these 3 metrics. Note that these metrics depend on a threshold for uncertainty, i.e., to define a prediction as certain or uncertain. Hence, they can be computed for different uncertainty thresholds. We plot the performance of all the baselines on these metrics for different uncertainty thresholds in Figure 3.
In addition, we report the Pascal VOC 2012 validation set accuracy and the runtime of a single forward pass for all the baselines in Table 1. Note that a single forward pass for the MC Dropout baseline consists of five stochastic forward passes and a single forward pass from the ensemble involves getting predictions from three ensemble components. Finally, we visualise the uncertainty estimates from each baseline for four samples from the Pascal VOC 2012 val set in Figure 4.
Observations: Firstly we note from   . Visualisation of different uncertainty baselines on samples from the PASCAL VOC validation set. The first column captures pixel-wise accuracy with bright signifying accurate and dark, inaccurate. The second and third columns show predictive entropy (PE) and mutual information (MI) obtained from the MC Dropout (MCDO) baseline respectively, the fourth and fifth columns show the PE and MI from deep ensembles. The sixth column maps per-pixel softmax entropy which is the aleatoric uncertainty estimate of DDU, and finally the seventh column shows feature density, which is the epistemic component captured by DDU. All the baselines save DDU density (last column on the right) capture uncertainty, i.e., the brighter, the more uncertain whereas DDU feature density captures confidence and hence brighter pixels signify more confident pixels and vice versa. maximum ambiguity or observation noise. On the other hand, epistemic uncertainty is high on regions which are previously unseen (or less seen) by the model. In the first two samples (first two rows), the epistemic uncertainty is not high and aleatoric uncertainty is captured on the edges by softmax entropy. In the last sample (last row), the epistemic uncertainty is high for a big patch which is inaccurately predicted. DDU feature density for that entire patch is significantly lower whereas softmax entropy doesn't capture it and is only high on the edges.

Conclusion
In this paper, we show that Deep Deterministic Uncertainty (DDU) can be easily extended to the task of semantic segmentation. We find that fitting DDU to a semantic segmentation model with a fully convolutional architecture can be done in a pixel-independent fashion, thereby making its adoption relatively simple. Finally, with experiments on Pascal VOC 2012 using DeepLab-v3+, we observe that DDU outperforms other well-known methods of uncertainty quantification without compromising on accuracy/mIoU and with the runtime of a single deterministic model.