{'fraction_non_alphanumeric': 0.030625279427732004, 'fraction_numerical': 0.012741904579421344, 'mean_word_length': 4.809833024118738, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation

Self-supervised monocular depth estimation has achieved impressive performance on outdoor datasets. Its performance however degrades notably in indoor environments because of the lack of textures. Without rich textures, the photometric consistency is too weak to train a good depth network. Inspired by the early works on indoor modeling, we leverage the structural regularities exhibited in indoor scenes, to train a better depth network. Specifically, we adopt two extra supervisory signals for self-supervised training: 1) the Manhattan normal constraint and 2) the co-planar constraint. The Manhattan normal constraint enforces the major surfaces (the floor, ceiling, and walls) to be aligned with dominant directions. The co-planar constraint states that the 3D points be well fitted by a plane if they are located within the same planar region. To generate the supervisory signals, we adopt two components to classify the major surface normal into dominant directions and detect the planar regions on the fly during training. As the predicted depth becomes more accurate after more training epochs, the supervisory signals also improve and in turn feedback to obtain a better depth model. Through extensive experiments on indoor benchmark datasets, the results show that our network outperforms the state-of-the-art methods. The source code is available at https://github.com/SJTU-ViSYS/StructDepth .


Introduction
Inferring the dense 3D map from a single image is a challenging problem without satisfactory solutions until the booming of deep neural networks. With the deep convolutional neural networks (CNNs), we can predict the accurate depth from a single image, via training the network with a lot of ground-truth depth labels. The recent selfsupervised learning paradigm does not require the groundtruth depth, while still obtaining high-quality results on benchmark datasets, using the photometric consistency as the major supervisory signal. Nevertheless, when existing self-supervised methods are trained on indoor images, the quality of depth estimation degrades notably [53] [3]. The main reason is the lack of textures in indoor images. Unlike outdoor scenes, the indoor scenes are full of texture-less regions, such as white walls, ceilings, and floors. Without rich textures, the photometric loss becomes too weak to train a good depth model. Seeking stronger or extra supervisory signals is therefore necessary for training a better depth network.
There have been a few attempts. An optical-flow field propagated from the sparse SURF [1] flow by a selfsupervised network, is used to guide training on textureless regions [53]. Another attempt [50] is to use an image patch instead of individual pixels to compute the photometric loss and apply extra constraints to the depth within the planar regions extracted from image segmentation. Though those attempts improve the results, they did not fully exploit the structural regularities presented in indoor environments, a valuable source of information for 3D learning. The structural regularities, known as the Manhattan-world model [6], describe that the scene consists of major planes aligned with dominant directions. This simple yet effective high-level prior leads to a much better performance in many vision tasks, such as indoor modeling [16][17] [5], visual SLAM [52] [12] [44], and visual-inertial odometry [56], but has not been applied to monocular depth learning.
In this work, we propose to apply the high-level prior of indoor structural regularities to self-supervised depth estimation as shown in Fig. 1. Specifically, we adopt two extra supervisory signals for training: 1) the Manhattan normal constraint and 2) the co-planar constraint. The Manhattan normal constraint enforces the major surfaces (the floor, ceiling, and walls) to be aligned with dominant directions. The co-planar constraint states that the 3D points be well fitted by a plane if they are located within the same planar region. We add two extra components into the training process. The first one is Manhattan normal detection. It classifies the major surface normal, computed from the depth predicted by the network, into the directions associated with the vanishing points by an adaptive thresholding scheme. The second one is planar region detection. We fuse the color and the geometric information derived from the depth and apply a classic segmentation algorithm to extract planar regions. During training, the two components incorporate the estimated depth to produce supervisory signals on the fly. Though those signals may be noisy in early epochs because of inaccurate depth, they will gradually improve as the depth quality improves, and in turn benefit the depth estimation.
We conduct experiments on the indoor benchmark datasets: NYU-v2 [40], ScanNet [7], and InteriorNet [29]. The results show that our method outperforms the existing state-of-the-art methods. Our main contributions are as follows: 1) A novel learning pipeline for self-supervised depth estimation leveraging structural regularities of indoor environments. To our best knowledge, this has not been presented in previous work.
2) Two novel components providing extra supervisory signals on the fly during the training process. Our components can be used to train a multi-task network including depth estimation, normal estimation, and planar region detection in a self-supervised manner, although the latter two tasks serve to train a better depth model in our current implementation.
3) We set a new state-of-the-art in self-supervised indoor depth estimation.

Related Work
Monocular depth estimation. Depth estimation from a single image is an ill-posed problem that is known as extremely hard to be solved. Since the pioneer works [10,9] employed the convolution neural networks (CNNs) to regress the depth directly, a lot of CNN-based monocular depth estimation methods have been proposed [32,26,24,43,15], producing impressively accurate results in benchmark datasets. Most of them are supervised methods that require the ground-truth depth data for training.
Self-supervised depth learning without the ground-truth depth has emerged as a promising alternative as acquiring the ground-truth depth at a large scale is challenging. The image appearance was firstly introduced in [19] to replace the ground-truth depth as the supervisory signal to train a depth network. One image in a stereo pair was warped to the other view by the predicted depth. The difference between the synthesized image and the real image, or the photometric error, is then used for supervision. The idea was further extended to monocular settings [54] [19]. By the careful design of network architectures [20], loss functions [39], and online refinement [4], self-supervised approaches obtain impressive results on benchmark datasets.
Despite achieving impressive performance on outdoor datasets, such as KITTI [18] and Make3D [37], existing selfsupervised methods perform poorly in indoor datasets. The reason is that the indoor scenes are full of texture-less regions, such as white walls and ceilings, making the photometric loss become too weak to supervise the depth learning. Zhou et al. [53] adopted an optical-flow-based training paradigm supervised by the flow field from an optical flow network, initialized from sparse SURF [1] correspondences. The recent work [50] employed the more discriminative patches instead of individual pixels to compute the photometric loss, and also applied the piece-wise planar prior to depth learning by assuming that the homogeneous-color regions are planar regions. Though their approaches improve the performance. They did not fully exploit the structural prior of the environments. In addition, the planar-region assumption in [50] does not hold for planes with the same color, e.g. mutually perpendicular white walls. It therefore leads to false planar regions deteriorating the depth model.
Planar region detection. Though powerful planar-region detectors [30][45] [51] have been proposed recently and have shown high-quality results in complex indoor images. Those CNN-based detectors require a huge number of plane labels for training and are not suited for the self-supervised learning scheme. Though detecting planes in the image is challenging, if the depth is available, this task becomes much easier [36] [23]. Here, we detect the planar regions using a classic graph-based segmentation approach [11] simi- The neural network to be trained to predict the depth from a single image. b) Manhattan normal detection: It classifies the surface normal estimated from depth prediction into dominant directions. c) Planar region detection: Both the color and geometric information are used to extract planar regions by a graph-based segmentation. The planar region detection is kept updated with the improved depth during training iterations. Two extra losses, Manhattan normal loss and co-planar loss, are used to train the network, as indicated by the red dot arrows. lar to [50], while employing the additional geometric information extracted from the depth estimated on the fly when training. Though the depth may not be precise initially, it will gradually improve as the training progresses such that the segmentation will improve as well. With the additional geometric information, our approach avoids false planar regions that are indistinguishable by colors and produces less over-segmentation on texture-rich planar regions.
Structural regularities in indoor environments. Indoor scenes exhibit strong structural regularities, which can be described as the "Manhattan world". Namely, the scene can be decomposed into major planes, where their normal vectors are mutually orthogonal. These structural regularities are valuable priors that have been applied to a wide range of indoor 3D vision tasks, such as vSLAM [52][12] [44], VIO [56], and mapping [16][17] [5]. In fact, exploiting the structural prior of indoor scenes was probably the only geometric way to infer the 3D information from a single image in early days [8] [27]. It is natural to think that structural regularities should also benefit the learning-based vision tasks in indoor environments.
Wang et al. [41] propose to use the vanishing points and lines to train a surface normal estimator which achieves the state-of-the-art performance. Our work adopts a similar spirit but differs from theirs in that our major task is depth estimation, where the surface normal is just an intermediate result that serves for better training. In addition, our depth network is trained in a fully self-supervised manner and does not require the line map as the extra input. To our best knowledge, our work is the first one incorporating the structural regularities of indoor environments into selfsupervised monocular depth estimation.

Method
Our self-supervised depth learning pipeline is illustrated in Fig. 2. It consists of three major components. The first one is the depth network, which takes a single image as the input and predicts a depth map. We use the same architecture as in [50] for the depth network. Based on the predicted depth, the other two components, Manhattan normal detection and planar region detection, are used to produce the supervisory signals leveraging the structural prior of indoor environments. Manhattan normal detection aligns the normal computed from the depth map with the dominant orientations, estimated from the vanishing points in the image. Planar region detection applies a graph-based segmentation to detect the planar regions with the combination of color, normal, and plane-to-origin distance information. Both Manhattan normal detection and planar region detection may be inaccurate in the initial training epochs, but they will improve in later epochs as the depth prediction becomes better. The improved supervisory signals lead to a better depth prediction as well.
In the following sections, we'll describe how we apply the Manhattan normal constraint and the co-planar constraint in our training process.

Manhattan normal constraint
Dominant direction extraction. The structural regularities of indoor environments imply that most indoor scenes contain planar surfaces aligned with dominant directions. The dominant directions can be estimated from the structural lines in the image. The intersection of a set of parallel structural lines in the image is the vanishing point. Let v be a vanishing point extracted from the 2D image. One of the dominant directions in the camera coordinate system is computed as where η ∈ R 3 is a unit vector representing this dominant direction and K is the camera intrinsic matrix. Note that we need only two vanishing points to get all the dominant directions, since the third dominant direction can be obtained by the cross product. We apply the 2-Line searching method [33] to extract the dominant directions from the image. The dominant direction extraction is done only once before training. Both the extracted directions and their reverse directions are considered to be the possible normal directions of the major planes in the scene, such as the ceiling, the floor, and the walls.
Surface normal estimation. To estimate the surface normal, we first get the 3D coordinates X p ∈ R 3 of each pixel p from the predicted depth by Here, D(p) denotes the depth predicted by the depth network. Next, we adopt a differentiable point-to-normal layer [46,47,22] to estimate the surface normal from the 3D points. Specifically, the normal n p of a given pixel p is calculated from a set of 3D points within a small neighborhood centering on point X p . The neighborhood is set as 7 × 7 in our implementation as the previous work [46]. Manhattan normal detection. Given the surface normal prediction n, we propose the Manhattan normal detection to classify the surface normal that belongs to the dominant planes. Our strategy is to compare the difference between the estimated normal vector n p and each dominant direction η k by using a cosine similarity s(·, ·) and choose the one with the best similarity, namely where n align p is the aligned normal and the cosine similarity is defined as s(n p , η k ) = (n p · η k )/( n p · η k ). Let the maximum similarity of each pixel be s max p . We define the Manhattan mask as: where 1 and 0 represent Manhattan and non-Manhattan regions respectively Figure 3: The pipeline of planar region detection. Both the color and geometric information are used to compute the dissimilarity for planar region segmentation. The color dissimilarity is calculated by comparing the RGB colors. The geometry dissimilarity is the sum of the normal and the plane-to-origin distance dissimilarities. Based on the proposed dissimilarity, a graph-based segmentation [11] is applied to extract the planar regions.
During the training, we use an adaptive thresholding scheme for detecting the Manhattan regions. We initially set a relatively small threshold to allow more pixels being classified into the Manhattan region because of inaccurate normal estimates, and gradually increase the threshold since the normal estimates become accurate in later epochs. In our implmentation, the threshold γ grows with the iteration number N train linearly: γ = α · N train + β, where α and β are set to 1.633e −3 and 0.9 respectively.
Manhattan normal loss. We apply the Manhattan normal constraint within the Manhattan region by using the aligned normal obtained in (3) as the supervisory signal. The constraint enforces the estimated normal to be as close to the aligned normal as possible, which is described by a loss function L norm : where N norm is the number of pixels located in Manhattan regions, and M P p indicates whether the pixel p locates in the planar regions, which we'll introduce how to detect them in the following section.

Co-planar constraint
Planar region detection. To enforce the co-planar constraint, we need to detect the piece-wise planar region correctly. Previous work [50] detects the planar regions by assuming the regions with homogeneous colors are planar. This simple strategy, however, usually leads to false detection or over-segmentation producing false supervisory signals. We propose a novel planar region detection method, as shown in Fig. 3, which integrates both the color and the Figure 4: The proposed planar region detection during training. From the left to the right columns: the input images, the groud-truth depth, the estimated depth, the dissimilarity map, and the planar regions detected by only colors [50] and our method based on the color and geometric information. First row: Two walls cannot be distinguished by colors, but can be separated by our method. Second row: The floor is over-segmented by using only colors but can be correctly detected by our method.
online updated geometry information to extract the planar areas more reliably.
The key idea is that we adopt a novel dissimilarity map in the following graph-based segmentation. This dissimilarity takes the color, normal, and the plane-to-origin distance into consideration. We use the aligned normal to derive the dissimilarity instead of the estimated normal since we found the latter is too noisy. Let the 3D coordinates of a pixel p to be X p . Suppose this 3D point lies in the plane where the normal is the aligned normal n align p . The plane-to-origin distance is computed as Let q be the adjacent pixel of p. The normal dissimilarity between them is defined as the Euclidean distance between the two vectors: Denoting the minimum and maximum dissimilarities among all the adjacent pixels by D max n , D min n respectively, we define a [·] operator to normalize the dissimilarity via The plane-to-origin distance dissimilarity is defined as The geometric dissimilarity combines the normalized version of the two dissimilarities as The color dissimilarity is computed as where I p , I q are the RGB colors. Finally, we get the dissimilarity combining both the color and geometric information by Based on the dissimilarity, we apply the graph-based segmentation [11] and filter out small areas to obtain the planar regions following [50]. The advantage of using such a dissimilarity definition can be seen in Fig. 4. Comparing with using only the color information, our method avoids false planar regions that cannot be distinguished by colors and also over-segmentation caused by different colors. Note that our planar region segmentation is be updated during training. As the training progresses, the gradually improved depth leads to better segmentation and vise versa.
Generate the co-planar depth. After detection of planar regions, we invoke the co-planar constraint to flatten the 3D points located within those plane regions. The first step is plane fitting for 3D points within the planar region. We obtain the plane parameters θ = −n/d ∈ R 3 as previous work [28,50] by solving the least squares problem where each column of X ∈ R 3×N represents a 3D point within the planar region. After that, the inverse depth ρ p of the pixel p by plane fitting is computed as where K represents the camera intrinsic matrix. We then transform the inverse depth to the depth D plane p with the maximum and minimum protection following [19,20,50].
Co-planar loss. The depth D plane p obtained from plane fitting is then used as an extra signal to constrain the estimated depth. The loss function is defined as  The first two blocks list the results of supervised methods. The second block contains the supervised methods with plane detection. The third and fourth blocks list the results of self-supervised methods. ↓ indicates the lower the better, ↑ indicates the higher the better. Our approach performs best among the self-supervised ones.
√ -supervised learning × -self-supervised learning pp -with post processing as in [19]  where N plane is the number of pixels within the planar regions M P .

Total loss
We use the image patches instead of individual pixels to compute the photometric loss as suggested in [50], which is defined as the combination of L1 loss and a structure similarity loss SSIM [55]: where N p denotes the local window surrounding p. ω is the relative weight of two parts and set as 0.85 the same as previous work [20]. We also adopt the edge-aware smoothness loss where ρ t ← ρ t /ρ t is the mean-normalized inverse depth, and ∂ x , ∂ y are the gradients along the x and y directions.

Experimental results
We train our model on the NYUv2 dataset [40] using the data split the same as the previous work [53] [50], and evaluate our methods on NYUv2 [40], ScanNet [7], and InteriorNet [29] datasets. We detect the vanishing points on the training images and skip 18 image sequences that fail to detect valid vanishing points. This results in 21465 monocular training sequences and 654 images for validation. Each monocular training sequence consists of five frames. Our network model adopts the same architecture as [50].
We compare our method with the state-of-the-art methods of monocular depth estimation. Apart from depth estimation, we also evaluate the performance of surface normal estimation, and present ablation studies about the effectiveness of the proposed supervisory signals, and using different network architectures. More results can be found in the supplementary material.

Implementation details
The network is trained for a total of 50 epochs with a batch size of 32 based on the pre-trained model [50]. We use Adam optimizer and a multi-step learning rate reduction strategy. We set the initial learning rate as 10 −4 , then decay it by 0.1 at the 26th epoch and 36th epoch. We perform random flipping and color augmentation during training. All images are firstly undistorted and cropped by 16 pixels from the border, and then scaled to 288 × 384 for training. The camera intrinsic parameters come from the official specification [40], and are adjusted to be consistent with the image cropping and scaling. We follow the same criteria used in [20,50] for evaluation. Namely, we cap the depth to 10m and use the median scaling strategy to avoid the scale ambiguity of monocular depth estimation. The evaluation metrics include root mean squared error (RMS), absolute relative error (AbsRel), mean log10 error (Log10), and the accuracy under threshold (δ i < 1.25 i , i = 1, 2, 3).

Results on NYUv2 Dataset
Depth estimation. The quantitative results of depth estimation are listed in Tab. 1. The results show that our method outperforms MovingIndoor [53] and P 2 Net [50], the state-of-the-art self-supervised methods on indoor monocular depth estimation, by a large margin. The results also show that our method surpasses some supervised approaches. The depth estimation results are visualized in Fig.  5. We can see that our method obtains more accurate indoor structures and smoother planes than existing methods.
Surface normal estimation. We also evaluate the surface normal estimation as shown in Tab. 2. Our method outperforms existing methods, and also some supervised methods [13,35,15]. Results are also shown in Fig. 5.    We use the model trained only on NYUv2 to evaluate our methods generalized to other indoor datasets. ScanNet [7] is captured with a depth camera attached to a iPad, containing around 2.5M RGBD video captured in 1513 scenes. We use the test split proposed by [50] which includes 533 images. The evaluation results are shown in Tab. 3 and Fig.  6. InteriorNet [29] is a synthetic dataset of indoor video sequences containing millions of well-designed interior design layouts, furniture and object models. Because there is no current official train/test split on InteriorNet for depth estimation, here we selected 540 images randomly from the HD7 data of the full dataset as test images. The evaluation results are shown in Tab. 4 and Fig. 7.

Results on ScanNet and InteriorNet
Although ScanNet and InteriorNet have not been used for training, the results show that our method still generalizes well and outperforms existing methods.    Table 5: Ablation study about using different supervisory signals. We evaluate the performances using only the Manhattan normal constraint (Normal-only), using only the coplanar constraint (Coplanar-only), and the proposed method (Our(full)). We also present the result of fine-tuned P 2 Net model (P 2 Net-finetune). Note all the models were trained with the same number of epochs for fair comparison.

Ablation study
To better understand the effectiveness of each part of our method, we perform an ablation study by changing various  We also test our method using different network architectures. As shown in Tab. 6, using the proposed supervisory signals, both models are improved, indicating our method is universal to different network architectures. But the results based on Monodepth2 are worse than those based on P 2 Net. This is largely due to the patch-based photometric loss that is better for texture-less regions as suggested in [50].

Planar-region detection in training
We show the intermediate planar region detection results during training in Fig. 8. The results show that the planar region segmentation gradually improves with the updated depth and normal estimates. By contrast, the color-only method produces false planar regions as indicated by the red rectangles.

Limitation
We discuss the limitations of our method. The first limitation is that extracting dominant directions highly relies on the Manhattan world assumption. It may not work well in indoor scenes with irregular layouts containing slant planes. Possible solutions include using a relaxed version of Manhattan world assumption as in [38] [56], or directly using the estimated direction from each detected vanishing point to derive the normal constraint. In other words, those dominant directions are not restricted to be mutually perpendicular. The second limitation is that the low quality of initial depth should be avoided. As our planar region detection relies on depth information, the low depth quality will deteriorate the segmentation results and generate false supervisory signals, which in turn prevent the network from converging to a good model. Our solution is to use a pre-trained depth model or train the model only with photometric and Figure 8: First row: The planar regions detected by the color-only method [50]. Bottom rows: The estimated depth, surface normal and segmentation results at different epochs on NYUv2. Our segmentation results gradually improve as the training progresses. smoothness losses in early epochs. It leaves open to design a better planar region detector given low-quality initial depth estimates.

Conclusion
In this paper, we propose to leverage the structural regularities of indoor environments for monocular depth estimation. Two extra losses, Manhattan normal loss and coplanar loss, are used to supervise the depth learning. Those supervisory signals are generated on the fly during training by Manhattan normal detection and planar region detection. Our method achieves the state-of-the-art result on indoor benchmark datasets.

Supplementary Material
Here, we present extra experimental results, including the additional visual results of our method, the outdoor tests, and the plane quality tests.

Extra qualitative results
We include additional qualitative results on NYUv2, ScanNet, and InteriorNet datasets. Fig. S2 shows the 3D structure recovered from the estimated depth. Fig. S3 and Fig. S4 illustrate the results of the depth and surface normal estimation. Those results show that our method achieves more accurate depth estimation and produces more accurate 3D structures, compared with the existing methods.

Outdoor tests
We present the results of our method on the KITTI dataset, which is captured in outdoor scenes. We use the split composed of 44234 images as the training dataset, the same as Monodepth2 [20]. We firstly detected the vanishing points on the training images and skipped 335 images which fail to detect valid vanishing points. Consequently, 39500 image sequences were used for training and 4397 image sequences for validation. Other dataset preprocessing settings are consistent with [20]. The total epoch number of training is 17 with a batch size of 16. The initial learning rate is 10 −5 and drops to 10 −6 after 15 epochs. Results are shown in Tab [20] architecture achieves better performance than using P 2 Net. This is largely due to that the outdoor environments are full of textures. The well-designed Mon-odepth2 works well in such kinds of scenes, while the strategies adopted in P 2 Net are more suitable for indoor scenes. This has been discussed in [50]. Though our method does not improve the performance too much by using Mon-odepth2 architecture, we can see the effectiveness of our extra structural losses by using the P 2 Net architecture. The depth, the surface normal, and the plane detection results of our method on KITTI dataset are shown in Fig.  S1. Note that the detected planar regions are mostly located on the road, where the textures are rich enough to supervise a good depth. This may be the major reason why our extra losses did not help too much within the Monodepth2 training pipeline. The other reason may be that the extracted dominant directions may not be strictly mutually perpendicular in outdoor scenes, leading to large surface normal errors.

Plane quality tests
We evaluate the plane quality on the IBims-1 [25] as [49]. All models are trained on the NYUv2 dataset with the same number of epochs for fair comparison (pretrain epochs are also included in our method). From the results in Tab. 2, as expected, our method produces the best plane quality (the second column), even though P2Net also adopts co-planar losses. The improvements are largely due to the global constraint from Manhattan normal loss. However, all methods produce low structure quality especially the depth edge comparing with supervised methods, indicating great efforts are still required to improve self-supervised depth learning in indoor scenes.   Figure S1: Visualization of the KITTI results. From top to bottom rows: the input image, the estimated depth, the aligned surface normal, and the planar regions detected by our method based on the color and geometric information. Figure S2: Point cloud visualization on NYUv2, ScanNet and InteriorNet results. We present the point cloud results of Monodepth2 [20], P 2 Net [50], our method, and the ground-truth. We draw the dominant directions in the scene for better comparison. The results show that our method produces more accurate 3D structures.  [20], P 2 Net [50], our method, and the ground-truth depth / normal are presented for comparison. Compared with P 2 Net [50] and Monodepth2 [20], our method obtains better surface normal estimation and depth prediction as indicated by the red rectangles.  [20], P 2 Net [50], ours and the ground-truth depth / normal are presented for comparison. Compared with P 2 Net [50] and Monodepth2 [20], our method obtains better surface normal estimation and depth prediction as indicated by the red rectangles. Our models were trained on the NYUv2 dataset.