{'fraction_non_alphanumeric': 0.031158436361037663, 'fraction_numerical': 0.010826841992424783, 'mean_word_length': 5.029082292115468, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised Representation Learning

In the current monocular depth research, the dominant approach is to employ unsupervised training on large datasets, driven by warped photometric consistency. Such approaches lack robustness and are unable to generalize to challenging domains such as nighttime scenes or adverse weather conditions where assumptions about photometric consistency break down. We propose DeFeat-Net (Depth&Feature network), an approach to simultaneously learn a cross-domain dense feature representation, alongside a robust depth-estimation framework based on warped feature consistency. The resulting feature representation is learned in an unsupervised manner with no explicit ground-truth correspondences required. We show that within a single domain, our technique is comparable to both the current state of the art in monocular depth estimation and supervised feature representation learning. However, by simultaneously learning features, depth and motion, our technique is able to generalize to challenging domains, allowing DeFeat-Net to outperform the current state-of-the-art with around 10% reduction in all error measures on more challenging sequences such as nighttime driving.


Introduction
Recently there have been many advances in computer vision tasks related to autonomous vehicles, including monocular depth estimation [22,83,73] and feature learning [13,61,65]. However, as shown in Figure 1, these approaches tend to fail in the most complex scenarios, namely adverse weather and nighttime conditions.
In the case of depth estimation, this is usually due to the assumption of photometric consistency, which starts to break down in dimly-lit environments. Feature learning can overcome such strong photometric assumptions, but these approaches tend to require ground truth pixel-wise correspondences and obtaining this ground truth in crossseasonal situations is non-trivial. Inconsistencies between GPS measurements and drift from Visual Odometry (VO) makes automatic pointcloud alignment highly inaccurate and manual annotation is costly and time-consuming.
We make the observation that depth estimation and feature representation are inherently complementary. The process of estimating the depth for a scene also allows for the computation of ground-truth feature matches between any views of the scene. Meanwhile robust feature spaces are necessary in order to create reliable depth-estimation systems with invariance to lighting and appearance change.
Despite this relationship, all existing approaches tackle these challenges independently. Instead, we propose DeFeat-Net, a system that is capable of jointly learning depth from a single image in addition to a dense feature representation of the world and ego-motion between consecutive frames. What's more, this is achieved in an entirely self-supervised fashion, requiring no ground truth other than a monocular stream of images.
We show how the proposed framework can use the existing relationships between these tasks to complement each other and boost performance in complex environments. As has become commonplace [23], the predicted depth and ego-motion can be used to generate a correspondence map between consecutive images, allowing for the use of photometric error based losses. However, these correspondences can also be used as positive examples in relative metric learning losses [65]. In turn, the learnt features can provide a more robust loss in cases where photometric errors fail, i.e. nighttime conditions. The remainder of the paper provides a more detailed description of the proposed DeFeat-Net framework in the context of previous work. We extensively show the benefits of our joint optimization approach, evaluating on a wide variety of datasets. Finally, we discuss the current state-of-theart and opportunities for future work. The contributions of this paper can be summarized as: 1. We introduce a framework capable of jointly and simultaneously learning monocular depth, dense feature representations and vehicle ego-motion.
2. This is achieved entirely self-supervised, eliminating the need for costly and unreliable ground truth data collection.
3. We show how the system provides robust depth and invariant features in all weather and lighting conditions, establishing new state-of-the-art performance.

Related Work
Here we review some of the most relevant previous work, namely in depth estimation and feature learning.

Depth Estimation
Traditionally, depth estimation relied on finding correspondences between every pixel in pairs of images. However, if the images have been stereo rectified, the problem can be reduced to a search for the best match along a single row in the target image, known as disparity estimation. Initial methods for disparity estimation relied on hand-crafted matching techniques based on the Sum of Squared Differences (SSD), smoothness and energy minimization.
Supervised. Ladickỳ [33] andŽbontar [79] showed how learning the matching function can drastically improve the performance of these systems. Mayer et al. [46] instead proposed DispNet, a Fully Convolutional Network (FCN) [40] capable of directly predicting the disparity map between two images, which was further extended by [50]. Kendall et al. [30] introduced GC-Net, where the disparity is processed as a matching cost-volume in a 3D convolutional network. PSMNet [9] and GA-Net [81] extended these cost-volume networks by introducing Spatial Pooling Pyramid (SPP) features and Local/Semi-Global aggregation layers, respectively. Estimating depth from a single image seemed like an impossible task without these disparity and perspective cues. However, Saxena [58] showed how it is possible to approximate the geometry of the world based on superpixel segmentation. Each superpixel's 3D position and orientation is estimated using a trained linear model and an MRF. Liu et al. [38,39] improve on this method by instead learning these models using a CNN, while Ladickỳ et al. [34] incorporate semantic information as an alternative cue.
Eigen et al. [14,15] introduced the first methods for monocular depth regression using end-to-end deep learning by using a scale-invariant loss. Laina [35] and Cao [7] instead treated the task of monocular estimation as a classification problem and introduced a more robust loss function. Meanwhile, Ummenhofer et al. [66] introduced DeMoN, jointly training monocular depth and egomotion in order to perform Structure-from-Motion (SfM). In this paper we go one step further, jointly learning depth, egomotion and the feature space used to support them.
Unsupervised -Stereo Training. In order to circumvent the need for costly ground truth training data, an increasing number of approaches have been proposed using photometric warp errors as a substitute. For instance, Deep-Stereo [17] synthesizes novel views using raw pixels from arbitrary nearby views. Deep3D [74] also performs novel view synthesis, but restricts this to stereo pairs and introduces a novel image reconstruction loss. Garg [18] and Godard [23] greatly improved the performance of these methods by introducing an additional autoencoder and left-right consistency losses, respectively. UnDeepVO [37] additionally learns monocular VO between consecutive frames by aligning the predicted depth pointclouds and enforcing consistency between both stereo streams. More recently, there have been several approaches making use of GANs [1,53]. Most notably, [62] uses GANs to perform day-night translation and provide an additional consistency to improve performance in nighttime conditions. However, the lack of any explicit feature learning makes it challenging to generalize across domains.
Unsupervised -Monocular Training. In order to learn unsupervised monocular depth without stereo information, it is necessary to learn a surrogate task that allows for the use of photometric warp losses. Zhou et al. [82,83] introduced some of the first methods to make use of VO estimation to warp the previous and next frames to reconstruct the target view. Zhan [80] later extended this by additionally incorporating a feature based warp loss. Babu et al. [3,44] proposed an unsupervised version of DeMoN [66]. Other published methods are based upon video processing with RNNs [69] and LSTMs [51] or additionally predicting scene motion [67] or optical flow [29,70,78].
The current state-of-the-art has been pushed by methods that incorporate additional constraints [68] such as temporal [45], semantic [10], edge & normal [75,76], cross-task [84] and cycle [52,73] consistencies. Godard et al. [22] expanded on these methods by incorporating information from the previous frame and using the minimum reprojection error in order to deal with occlusions. They also introduce an automasking process which removes stationary pixels in the target frame. However, they still compute photometric losses in the original RGB colourspace, making it challenging to learn across domains.

Feature Learning
Hand-Crafted. Initial approaches to feature description typically relied on heuristics based on intensity gradients in the image. Since these were computationally expensive, it became necessary to introduce methods capable of finding interesting points in the image, i.e. keypoints. Some of the most well-know methods include SIFT [41] and its variant RootSIFT [2], based on a Difference of Gaussians and Non-Maxima Suppression (NMS) for keypoint detection and HOG descriptors.
Research then focused on improving the speed of these systems. Such is the case with SURF [5], BRIEF [6] and BRISK [36]. ORB features [56] improved the accuracy, robustness and speed of BRIEF [6] and are still widely used.
Sparse Learning. Initial feature learning methods made use of decision trees [55], convex optimization [63] and evolutionary algorithms [31,32] in order to improve detection reliability and discriminative power. Intelligent Cost functions [24] took this a step further, by using Gaussian Processes to learn appropriate cost functions for optical/scene flow.
Since the widespread use of deep learning, several methods have been proposed to learn feature detection and/or description. Balntas et al. [4] introduced a method for learning feature descriptors using in-triplet hard negative mining. LIFT [77] proposes a sequential pipeline consisting of keypoint detection, orientation estimation and feature description, each performed by a separate network. LF-Net [49] builds on this idea, jointly generating dense score and orientation maps without requiring human supervision.
On the other hand, several approaches make use of networks with shared encoder parameters in order to simultaneously learn feature detection and description. Georgakis et al. [20] learn 3D interest points using a shared Fast R-CNN [21] encoder. Meanwhile, DeTone introduced Super-Point [12] where neither decoder has trainable parameters, improving the overall speed and computational cost. More recently, D2-Net [13] proposed a describe-then-detect approach where the network produces a dense feature map, from which keypoints are detected using NMS.
Dense Learning. Even though SuperPoint [12] and D2-Net [13] produce dense feature maps, they still focus on the detection of interest points and don't use their features in a dense manner. Weerasekera et al. [72] learn dense features in the context of SLAM by minimizing multi-view match-ing cost-volumes, whereas [60] use generative feature learning with scene completion as an auxiliary task to perform visual localisation.
The Universal Correspondence Network [11] uses optical correspondences to create a pixel-wise version of the contrastive loss. Schmidt [59] instead propose semisupervised training with correspondences obtained from KinectFusion [47] and DynamicFusion [48] models. Fathy [16] and Spencer [65] extended the pixel-wise contrastive loss to multiple scale features through a coarse-to-fine network and spatial negative mining, respectively. On the other hand, SDC-Net [61] focuses on the design of the network architecture, increasing the receptive field through stacked dilated convolution, and apply the learnt features to optical flow estimation.
In this work we attempt to unify state-of-the-art feature learning with monocular depth and odometry estimation. This is done in such a way that the pixel-wise correspondences from monocular depth estimation can support dense feature learning in the absence of ground-truth labels. Meanwhile, computing match-costs in the learned feature space greatly improves the robustness of the depth estimation in challenging cross-domain scenarios.

Methodology
The main objective of DeFeat-Net is to jointly learn monocular depth and dense features in order to provide more robust estimates in adverse weather conditions. By leveraging the synergy between both tasks we are able to do this in a fully self-supervised manner, requiring only a monocular stream of images. Furthermore, as a byproduct of the training losses, the system additionally learns to predict VO between consecutive frames. Figure 2 shows an overview of DeFeat-Net. Each training sample is composed of a target frame I t and a set of support frames I t+k , where k ∈ {−1, 1}. Using the predicted depth for I t and the predicted transforms to I t+k we can obtain a series of correspondences between these images, which in turn can be used in the photometric warp and pixel-wise contrastive losses. The code and pre-trained models for this technique will be available at https:// github.com/jspenmar/DeFeat-Net.

Networks
DispNet. Given a single input image, I t , its corresponding depth map is obtained through  This decoder also produces intermediate disparity maps at each stage, resulting in four different scales.
PoseNet. Similarly, the pose prediction network Φ P consists of a multi-image ResNet encoder, followed by a 4layer convolutional decoder. Formally, where P t→t+k is the predicted transform between the cameras at times t and t + k. As in [22,68] the predicted pose is composed of a rotation in axis-angle representation and a translation vector, scaled by a factor of 0.001.

FeatNet.
The final network produces a dense ndimensional feature map of the given input image, Φ F : N H×W ×3 → R H×W ×n . As such, we define the corresponding L2-normalized feature map as In this case, Φ F is composed of a residual block encoderdecoder with skip connections, where the final encoder stage is made up of an SPP [9] with four scales.

Correspondence Module
Using the predicted D t and P t→t+k we can obtain a set of pixel-wise correspondences between the target frame and each of the support frames. Given a 2D point in the image p and its homogeneous coordinatesṗ we can obtain its corresponding location q in the 3D world through where π −1 is the backprojection function, K t is the camera's intrinsics and D t (p) the depth value at the 2D pixel location estimated using (1).
We can then compute the corresponding point c t→t+k by projecting the resulting 3D point onto a new image with where P t→t+k is the transform to the new coordinate frame, i.e. the next or previous camera position from (2). Therefore, the final correspondences map is defined as These correspondences can now be used in order to determine the sampling locations for the photometric warp loss and the positive matches in a pixel-wise contrastive loss to learn an appropriate feature space.

Losses
Once again, it is worth noting that DeFeat-Net is entirely self-supervised. As such, the only ground truth inputs required are the orginal images and the camera's intrinsics. Pixel-wise Contrastive. In order to train Φ F , we make use of the well established pixel-wise contrastive loss [11,59,65]. Given two feature vectors from the dense feature maps, f 1 = F 1 (p 1 ) and f 2 = F 2 (p 2 ), the contrastive loss is defined as with y as the label indicating if the pair is a correspondence, d = ||f 1 − f 2 || and m the target margin between negative pairs. In this case, the set of positive correspondences is given by C t→t+k . Meanwhile, the negative examples are generated using one of the spatial negative mining techniques from [65]. From both sets, a label mask Y is created indicating if each possible pair of pixels is a positive, negative or should be ignored. As such, the final loss is defined as This loss serves to drive the learning of a dense feature space which enables matching regardless of weather and seasonal appearance variations. Photometric and Feature Warp. We also use the correspondences in a differentiable bilinear sampler [28] in order to generate the warped support frames and feature maps where is the sampling operator. The final warp losses are a weighted combination of SSIM [71] and L1, defined by The photometric loss L P serves primarily to support the early stages of training when the feature space is still being learned.
Smoothness. As an additional regularizing constraint, we incorporate a smoothness loss [27]. This enforces local smoothness in the predicted depths proportional to the strength of the edge in the original image, ∂I t . This is defined as where λ is a scaling factor typically set to 0.001. This loss is designed to avoid smoothing over edges by reducing the weighting in areas of strong intensity gradients.

Masking & Filtering
Some of the more recent improvements in monocular depth estimation have arisen from explicit edge-case handling [22]. This includes occlusion filtering and the masking of stationary pixels. We apply these automatic procedures to the correspondences used to train both the depth and dense features. Minimum Reprojection. As the camera capturing the monocular stream moves throughout the scene, various elements will become occluded and disoccluded. In terms of a photometric error based loss, this means that some of the correspondences generated by the system will be invalid. However, when multiple consecutive frames are being used, i.e. k ∈ {−1, 1}, different occlusions occur in each image.
By making the assumption that the photometric error will be greater in the case where an occlusion is present, we can filter these out by simply propagating the correspondence with the minimum error. This is defined as Due to the nature of the training method and implicit depth priors (i.e. regions further away change less) stationary frames or moving objects can cause holes of infinite depth in the predicted depth maps. An automasking procedure is used to remove these stationary pixels from contributing to the loss, where µ is the resulting mask indicating if a correspondence is valid or not and [] is the Iverson bracket. In other words, pixels that exhibit lower photometric error to the unwarped frame than to the warped frame are masked from the cost function.

Results
Each subsystem in DeFeat-Net follows a U-Net structure with a ResNet18 encoder pretrained on ImageNet, followed by a 7 layer convolutional decoder similar to [23]. The code and pre-trained models will be available at https: //github.com/jspenmar/DeFeat-Net. In all our experiments, the warp loss parameter is set to α = 0.85 as per [28].
On the KITTI dataset [19] we follow the Eigen-Zhou evaluation protocol of [23,83]. This dataset split provides 39,810 training images and 4,424 validation images. These images are all from a single domain (sunny daytime driving).
We also make use of the RobotCar Seasons dataset [57]. This is a curated subset of the larger RobotCar dataset [43], containing 49 sequences. The dataset was specifically chosen to cover a wide variety of seasons and weather conditions, leading to greater diversity in appearance than KITTI.
Unlike the KITTI dataset, which provides sparse groundtruth depth from LiDAR, RobotCar Seasons does not include any depth ground truth. Our proposed technique is unsupervised, and can still be trained on this varied dataset, but the lack of ground truth makes quantitative evaluation on RobotCar Seasons impossible. To resolve this, we returned to the original RobotCar dataset and manualy created a validation dataset comprising of 12,000 images with

Single Domain Evaluation
We first evaluate our approach on the KITTI dataset, which covers only a single domain. For evaluation of depth accuracy, we use the standard KITTI evaluation metrics, namely the absolute relative depth error (ABS REL), the relative square error (SQ REL) and the root mean square error (RMSE). For these measures, a lower number is better. We also include the inlier ratio measures (A1, A2 and A3) of [23] which measure the fraction of relative depth errors within 25%, 25 2 % and 25 3 % of the ground truth. For these measures, a larger fraction is better.
To evaluate the quality of the learned feature representations, we follow the protocol of [65]. We compute the average distance in the feature space for the positive pairs from the ground-truth (µ + ), and the negative pairs (µ − ). Naturally a smaller distance between positive pairs, and a larger distance between negative pairs, is best. We also compute the Area Under the Curve (AUC) which can be interpreted as the probability that a randomly chosen negative sample will have a larger distance than the corresponding positive ground truth match. Therefore, higher numbers are better. Following [65] all three errors are split into both local (within 25 pixels) and global measurements.
The results of the depth evaluation are shown in Table 1 and the feature evaluation is shown in Table 2. We can see that in this single-domain scenario, the performance of our technique is competitive with MonodepthV2 and clearly outperforms most other state-of-the-art techniques for monocular depth estimation. The results for [22] were obtained by training a network using the code provided by the authors.
Regarding the features, L2 denotes the L2-normalized versions, whereas G, L & GL represent the different negative mining variants from [65]. We can also see that despite being unsupervised, our learned feature space is competitive with contemporary supervised feature learning techniques and greatly outperforms pretrained features when evaluating locally. It is interesting, however, to note that the simple act of L2-normalizing can improve the global performance of the pretrained features.
Our feature space tends to perform better in the Global evaluation metrics than the local ones. This is unsurprising as the negative samples for the contrastive loss in (7) are obtained globally across the entire image.

Multi-Domain Evaluation
However, performance in the more challenging Robot-Car Seasons dataset demonstrates the real strength of jointly learning both depth and feature representations. RobotCar Seasons covers multiple domains, where traditional photometric based monocular depth algorithms struggle and where a lack of cross-domain ground-truth has historically made feature learning a challenge. For this evaluation, we select the best competing approach from Table 1 (MonodepthV2) and retrain both it and DeFeat-Net on the RobotCar Seasons dataset. All techniques are trained from scratch.
The results are shown in Table 3 and example depth map comparisons are shown in Figure 3. We can see that in this more challenging task, the proposed approach outperforms the previous state of the art technique across all error measures. While for the daytime scenario, the improvements are modest, on the nighttime data there is a significant improvement with around 10% reduction in all error measures.
We believe that the main reason behind this difference is  that in well-lit conditions, the photometric loss is already a good supervision signal. In this case, incorporating the feature learning adds to the complexity of the task. However, nighttime scenarios make photometric matching less discriminative, leading to weaker supervision. Feature learning provides the much needed invariance and robustness to the loss, leading to the significant increase in performance.
It is interesting to note that the proposed approach is especially robust with regards to the number of estimated outliers. The A1, A2 and A3 error measures are fairly consistent between the day and night scenarios for the proposed technique. This indicates that even in areas of uncertain depth (due to under-exposure and over-saturation), the proposed technique fails gracefully rather than producing catastrophically incorrect estimates.
Since previous state-of-the-art representations cannot be trained unsupervised, and RobotCar Seasons does not provide any ground-truth depth, it is not possible to repeat the feature comparison from Table 2 in the multi-domain scenario. Instead Figure 4 compares qualitative examples of the learned feature spaces. For these visualizations, we find the linear projection that best shows the correlation between the feature map and the images and map it to the RGB color cube. This dimensionality reduction removes a significant amount of discriminative power from the descriptors, but allows for some form of visualization.
In all cases, the feature descriptors can clearly distinguish scene structures such as the road. It is interesting to note that a significant degree of context has been encoded in the features, and they are capable of easily distinguishing a patch in the middle of the road, from one on the left or right, and from a patch of similarly colored pavement. The feature maps trained on the single domain KITTI dataset can sometimes display more contrast than those trained on Robot-Car Seasons. Although this implies a greater degree of discrimination between different image regions, this is likely because the latter representation can cover a much broader range of appearances from other domains. Regarding, the nighttime features, it is interesting that those trained on a single domain seem to exhibit strange behaviour around external light sources such as the lampposts, traffic lights and headlights. This is likely due to the bias in the training data, with overall brighter image content.

Ablation
Finally, for each dataset we explore the benefits of concurrent feature learning, by re-training with the FeatNet subsystem disabled. As shown in Table 4, the removal of   Table 4. Performance with and without concurrent feature learning, on each dataset the concurrent feature learning from our technique causes a small and inconsistent change on the KITTI and RobotCar Day data. However, on the RobotCar Night data, our full approach drastically outperforms the version which does not learn a specialist matching representation. For many error measures, the performance doubles in these challenging scenarios, and the reduction in outliers causes a three-fold reduction in the Sq-Rel error. These findings reinforce the observation that the frequently used photometric warping loss is insufficient for estimating depth in challenging real-world domains.

Conclusions & Future Work
This paper proposed DeFeat-Net, a unified framework for learning robust monocular depth estimation and dense feature representations. Unlike previous techniques, the system is able to function over a wide range of appearance domains, and can perform feature representation learning with no explicit ground truth. This idea of co-training an unsupervised feature representations has potential applications in many areas of computer vision beyond monocular depth estimation.
The main limitation of the current approach is that there is no way to enforce feature consistency across seasons. Although depth estimation and feature matching work robustly within any given season, it is currently unclear weather feature matching between different seasons is possible. It would be interesting in the future to explore crossdomain consistency as an additional training constraint. However, this will necessitate the collection of new datasets with cross seasonal alignments.