{'fraction_non_alphanumeric': 0.037746023186842816, 'fraction_numerical': 0.01494113417812528, 'mean_word_length': 4.98158849616987, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth

Existing monocular depth estimation methods have achieved excellent robustness in diverse scenes, but they can only retrieve affine-invariant depth, up to an unknown scale and shift. However, in some video-based scenarios such as video depth estimation and 3D scene reconstruction from a video, the unknown scale and shift residing in per-frame prediction may cause the depth inconsistency. To solve this problem, we propose a locally weighted linear regression method to recover the scale and shift with very sparse anchor points, which ensures the scale consistency along consecutive frames. Extensive experiments show that our method can boost the performance of existing state-of-the-art approaches by 50% at most over several zero-shot benchmarks. Besides, we merge over 6.3 million RGBD images to train strong and robust depth models. Our produced ResNet50-backbone model even outperforms the state-of-the-art DPT ViT-Large model. Combining with geometry-based reconstruction methods, we formulate a new dense 3D scene reconstruction pipeline, which benefits from both the scale consistency of sparse points and the robustness of monocular methods. By performing the simple per-frame prediction over a video, the accurate 3D scene shape can be recovered.

How to ensure the depth consistency over consecutive frames is receiving increasing attention. Bian et al. [10], [24] employs an unsupervised paradigm and models the predicted depth prediction as scale-invariant. They propose a geometric consistency loss to implicitly learn the scale consistency over consecutive frames. Similarly, CVD [27] employs an unsupervised method but does the inference training to ensure consistency. By contrast, RCVD [28] takes the MiDaS model as the depth prior and estimates consistent dense depth maps and camera poses from a monocular video. They have achieved promising visual consistency of depth maps. However, we obverse that their reconstructed point cloud is still far from satisfactory. More detailed analysis can be shown in Section 3.3 and Figure 4. In this work, instead of the visual consistency, we focus on geometric consistency, i.e., achieving the 3D scene reconstruction from consecutive frames.
Following previous methods, we enforce the model to predict an affine-invariant depth, thus recovering scale and shift for the prediction is the main barrier for 3D scene reconstruction over a monocular video. Existing methods [1], [2], [3], [6] propose to directly compute a scale and shift value from least-squares fitting with the ground truth (GT), i.e. global recovery strategy. However, we observe that the optimal scale and shift are always heteroscedastic. During practical application, the global fitting does not consider the distribution difference between affine-invariant depth and ground-truth depth, and fails to effectively align the local regions. In Figure 5(a), we show an example error map visualization between the ground truth and such global scaleshift recovered depth, and there exists a low-frequency error clearly. Such a coarse alignment method cannot recover a high-quality metric depth for reconstruction.
Motivated by this observation, we propose a local recovery strategy to recover the locally aligned metric depth. Concretely, we employ the locally weighted linear regression as a novel metric depth alignment method, and demonstrate that heteroscedastic scale and shift maps can be recovered with as few as 25 points by enforcing spatial smoothness. Rather than fitting a unified scale map and shift map, i.e., sharing the same scale and shift values over all coordinates of the image, our local recovery strategy can retrieve a location-related scale map and shift map to adjust the distribution of prediction. Experiments show that our method can significantly improve metric accuracy. Although some sparse anchor points are required, compared with existing state-of-the-art depth completion methods [29], [30], [31], [32], which generally take more than hundreds to thousands of sparse points, our strategy requires much fewer points than them. Also experimentally we show that although completion methods have taken sparse depth measurements as inputs, our method can further boost their performance with our local recovery strategy.
Besides boosting performance, the second benefit of our local recovery strategy is to better analyze the weakness of all existing depth estimation methods, and guide the design and choice of loss functions. The depth error can be decoupled into two parts: the coarse misalignment error and the detail-missing error. Compared with the error of global fitting, the alleviated error achieved by the local recovery is the coarse misalignment error, while the remaining one is the detail-missing error. Through our re-local-alignment analytical experiments in Table 5, we observe that current state-of-the-art depth estimation methods, including supervised learning metric depth, unsupervised learning scaleinvariant depth, supervised learning affine-invariant depth, and depth completion, all suffer from noticeable misalignment issue w.r.t. the ground truth.
To achieve 3D scene reconstruction from a monocular video, retrieving a strong and robust monocular depth prediction model is also essential. We collect over 6.3 million images from the existing RGB-D datasets for training models using backbones such as ResNet50 [33] and Swin-L [34], and investigate that how much accuracy can we benefit from a large-scale dataset.
The last challenge is how to obtain accurate sparse anchor points as the metric guidance. For analytical evaluation, we leverage the ground-truth depth to decouple and analyze the composition of errors. For practical application, aligning with ground-truth depth can be the upper bound of our local recovery strategy. We perturb the ground truth manually and analyze the error to simulate the inaccuracy of sparse anchor points. Also, The SfM [35] can be employed to recover sparse depth points on distinguishable feature points in practice. Through per-frame alignment with such accurate guidance, we can achieve geometrically consistent depth and perform robust 3D scene reconstruction. Although existing geometry-based methods, e.g., multi-view stereo reconstruction [36], [37], [38], also takes the similar paradigms of structure from motion (SfM), their performance may suffer from inaccurate correspondences on lowtexture regions. By contrast, our per-frame prediction comes from our well-trained strong monocular depth estimation model, which is much more robust in the low-texture regions.
Finally, through applying the local recovery strategy, our ResNet50 model gains more than 50% improvement at most on current affine-invariant depth evaluation benchmarks. For 3D scene reconstruction, our pipeline significantly im-proves both the accuracy and consistency, and achieves better performance than related works on five NYU [18] videos. To summarize, our main contributions are as follows: • We propose a novel and effective metric depth recovery strategy, i.e., locally weighted linear regression, which significantly improves the accuracy of the recovered metric depth with a very sparse set of anchor points. Extensive experiments show that our method can lead to up to 50% improvements in performance over compared state-of-the-art methods.
• Our local recovery strategy can be an analytical tool for subsequent depth prediction work, enabling decoupling the prediction errors and analyzing the weakness of their models. • We train a robust monocular depth estimation model on a large diverse dataset that contains 6.3 million images in total. We provide detailed analysis of its performance w.r.t. the training dataset size using our analytical tool.
• Aiming at the video-based scenarios, by combining our strong monocular depth estimation model with a geometry-based method for retrieving highconfidence anchor points, we design a new pipeline for robust and dense 3D scene reconstruction.

RELATED WORK
Single Image Depth Estimation. Single image depth estimation is an important problem. Existing methods [8], [39], [40] have achieved promising results. Typically, the resulting models generalize poorly to diverse scenes due to the fact that almost all of these methods train the model on a moderate-size dataset. To mitigate this problem, Chen et al. [9] constructed the first large-scale and diverse dataset (DIW), and enforced the network to learn depth ordering relations, namely relative depth. The DIW dataset only annotates a pair of points in each image. Several approaches [4], [5], [7], [9], [41] obtained the ground truth of dense relative depth by leveraging online web-stereo videos or images. Owing to large-scale and diverse data samples, these strategies can produce a model with stronger generalization.
Although learning relative depth can obtain a robust model, the relative depth can only represent ordinal depth relations, i.e., one point is farther or closer than another one, and loses much of the geometry information. Therefore, some work [1], [3], [6], [8] proposed to learn the affineinvariant depth. Ranftl et al. [3] proposed the scale-shift invariant loss to leverage the training on multi-source data, achieving promising generalization on diverse scenes. Yin et al. [1] employ a heterogeneous loss training strategy, which obtains state-of-the-art performance on multiple zero-shot testing datasets. Our method is built upon these robust depth estimation methods using multiple RGB-D datasets for the strong generalization capability. We also go one step further to remedy the issue of scale-missing by devising a powerful local scale recovery technique by exploiting additional easy-access information. RGB-D Datasets. Datasets [18], [42], [43], [44] are significant for the advancement of data-driven depth prediction methods. Current datasets can be summarized into two categories according to their quality. Some datasets are captured by RGB-D sensors, which can retrieve accurate metric depth. Make3D [42] is the first outdoor RGB-D dataset constructed for monocular depth prediction study. KITTI and NYU are captured by LIDAR on outdoor streets and Kinect in indoor rooms. Larger-scale RGB-D datasets are also constructed, such as ScanNet [43], Taskonomy [45], DIML [46], and DIODE [44]. These datasets usually contain very limited scenes only.
To cover more diversified scenes, some algorithms try to leverage online images and videos. MegaDepth [5] employs structure from motion to construct the depth supervision on the still and rigid scenes. To include more non-rigid and diverse scenes, Xian et al. [41] and Wang et al. [47] adopted optical flow techniques to build datasets of relative depth. Yin et al. [6], [8] propose another large-scale and diverse RGB-D datasets: DiverseDepth. Depth Completion. Depth completion aims to recover a dense depth map from a very sparse sensor-captured depth. Some methods [29], [32], [48], [49], [50] combine the RGB texture information with sparse depth together to solve the problem. Commodity-level RGB-D cameras such as Kinect, RealSense, and Tango produce depth images that are semidense but can miss values on textureless regions. Several methods solve this problem as a depth inpainting task and leverage smoothness priors [51], background surface extrapolation [52], and surface normals [53]. Compared to depth completion, our method leverages the robust prior of monocular depth estimation and sparse points for fitting rather than inferring the depth map directly, which aims to adopt sparse easy-access depth and generalizes well to various scenarios. Consistent Video-based Depth Estimation. Consistent video-based depth estimation [27], [28], [54] has attracted extensive attention recently. These methods leverage robust single-view depth estimation, ensure the correspondences and geometric constraints across images, and achieve visually consistent video depth. CVD [27] proposed the inference training by fine tuning the depth model on the specific testing video. RCVD [28] learns a depth deformation map, optimizes pose and depth jointly, and fixes details with geometric constraints to ensure the depth consistency between frames. In contrast, our method not only ensures visual depth consistency, but also reserves the inherent geometric accuracy, leading to the dense, accurate and consistent 3D scene reconstruction. MVS-based Depth Estimation. MVS-based depth estimation approaches [36], [37], [38], [55], [56], [57], [58] attempted to improve the traditional structure-from-motion [35] and multi-view stereo [59] pipeline with a differentiable cost volume constructed by warping features to predict depth values. These methods have achieved promising results. However, they may fail to handle dynamic objects, lowtexture regions, occlusion and lighting changes. We instead leverage the robust monocular depth estimation model to perform per-frame prediction and align with the sparse but accurate anchor points obtained from multi-view methods, without suffering from the problems mentioned above.

PRELIMINARIES
As described in Section 1, existing robust monocular depth estimation methods tend to predict an affine-invariant depth, and the unknown scale and shift need to be recovered: d * = ds + µ, where d, s, and µ are the predicted affineinvariant depth, scale, and shift respectively. We first show the influence of incorrect shift to 3D point cloud unprojected from a single image in Section 3.1. Then, we analyze the duplicated and distorted 3D reconstruction caused by inconsistent scale or inconsistent shift along consecutive video frames in Section 3.2. Finally, we theoretically and qualitatively compare the current 3D scene reconstruction methods in Section 3.3.

Shape Distortions Caused by the Unknown Shift
The previous work LeReS [1] has demonstrated that the incorrect shift of depth maps can cause noticeable 3D shape distortion during reconstruction. We review the problem here by taking a pinhole camera model as an example. The unprojection equation is as follows.
where (x c , y c , z c ) are the 3D coordinates in the camera coordinate system, (u, v) are the corresponding 2D coordinates in an image, (u 0 , v 0 ) are the optical center of the pinhole camera, f is the focal length, and d is the depth of (u, v).
Assuming that a shift ∆d is appended to d, i.e., d = d + ∆d, the corresponding 3D coordinates are Here (x c , y c , z c ) can be disturbed by different degrees, leading to the 3D distortions, as shown in Figure 1.

RGB Recovered Shift
Negatively Disturbed Shift Positively Disturbed Shift Figure 1: The point clouds reconstructed from an RGB image with recovered shift depth and disturbed shift depth separately. Existing robust monocular depth estimation usually predict affine-invariant depth, whose unknown scale and shift values need to be recovered. With disturbed shift value, the unprojected point cloud loses geometry accuracy and is distorted in the 3D space.  [27], [28], [54] Visual consistency Loss of inherent depth geometry Unsupervised depth methods [10], [24], [60], [61], [62] Predicting scale-consistent depth and poses simultaneously Unsatisfactory depth accuracy MVS methods [36], [38], [55], [56] Robustness over various scenes Suffering in low-texture regions

Duplication Caused by Inconsistent Scale and Shift Along Frames
While performing 3D reconstruction on multiple frames, the camera coordinate system is transformed to the world coordinate system by: where R and T are the rotation and translation matrices from the camera to the world system. They are defined as follows.
where r ij and t ij stand for the i-th row and j-th column value of matrix R and T . For inconsistent scale or shift values between frames, x w , y w , z w will be scaled or shifted towards different directions, leading to the 3D shape duplication and distortion, as shown in Figure 2.

Analysis of 3D Reconstruction Approaches
To solve the 3D reconstruction from consecutive frames, many methods [24], [27], [28], [36] were proposed in the literature. We summarize them into the following categories.
Unsupervised depth estimation-based methods. Unsupervised depth estimation methods [10], [24], [60], [61], [62] armed with a pose network and a depth network for both relative pose and depth estimation. Bian et al. [24] have demonstrated that they can recover accurate poses and predict globally scale-consistent depth maps in most cases. Therefore, such methods can aggregate per-frame prediction together for 3D scene reconstruction theoretically. However, we observe that because of weak supervision from the photometric loss, these methods may not achieve satisfactory reconstruction (see SC-DepthV2 in Figure 4).

Consistent video depth estimation-based methods.
To improve the scale consistency and quality on the video, several methods [27], [63] propose to apply the inference training on videos. CVD [27] finetunes a well-trained monocular depth estimation network during testing to satisfy the geometric constraints. To solve the slow inference training issue, RCVD [28] further proposes to fix a strong single-image depth network (MiDaS [3]) and learn a depth deformation map by jointly optimizing the pose and depth over the video. Both methods can achieve promising visual consistency on videos. As shown in Figure 4, there is room to improve both methods.
Multi-view stereo-based methods. With multiple views, multi-view stereo-based methods [36], [38], [55], [56], [57], [58] can be used to achieve good 3D reconstruction. They achieve high-quality 3D reconstruction based on the dense feature matching along the epipolar lines. Note that these methods rely on feature point correspondence and can result in poor accuracy in low-texture image regions. Figure 4 shows that COLMAP [59] produces serious outliers on walls. The learning-based method DPSNet [56] can reduce the outliers but fail to handle inaccurate correspondences.
We summarize the aforementioned methods in Table 1 and compare them qualitatively in Figure 4. Unlike the video depth prediction methods, which aim to improve the visual consistency of depth, our work concentrates on geometric accuracy and consistency in 3D scene reconstruction. Compared to the unsupervised depth prediction methods, our work explicitly combines the monocular depth estimation method and structure-from-motion (SfM) methods together for robust and accurate depth prediction and 3D scene reconstruction. We obtain some sparse anchor points (used as ground truth) using SfM, which can provide valuable guidance. Compared to multi-view stereo methods, we employ a data-driven module (the monocular depth estimation part) to overcome the low-texture issues.  The pipeline for dense 3D scene reconstruction. The robust monocular depth estimation model trained on 6.3 million images, locally weighted linear regression strategy, and TSDF fusion [64] are the main components of our method. A sparse set of ground-truth depth points can be obtained with a SLAM system [65], or some SfM algorithms [35], [59].  [27] and SC-DepthV2 [24] suffer from inaccurate correspondences and weak supervision. RCVD [28] is affected by the inaccurate shift and tends to be distorted. Traditional MVS-based method COLMAP [35], [59] and learning-based MVS method DPSNet [56] can predict good depth maps, but fail in low-texture regions. Here, we propose to leverage sparse and accurate COLMAP points to recover the scale and shift maps with our local recovery strategy, calibrating the predicted depth maps.

Our Pipeline for Dense 3D Scene Reconstruction
The pipeline for our dense 3D scene reconstruction method is shown in Figure 3. Overall, our pipeline leverages the robust data-driven monocular depth estimation model, a small set of accurate anchor points, and RGB-D fusion.
Generally, a 2D RGB image can be unprojected to the 3D space with known camera intrinsic, pose, and depth map, as described in Equation 4. As the first two parameters can be obtained from camera calibration, IMU sensors or SfM algorithms individually, how to achieve an accurate and robust depth map occupies an important position. As described in Section 3.3, the unsupervised depth estimation methods may suffer from the weak supervision of photometric loss, and MVS-based methods can be affected by inaccurate correspondences. Therefore, the monocular depth estimation model trained on large diverse datasets becomes a good choice, which has achieved excellent robustness and accuracy on diverse scenes.
We train our robust monocular depth estimation models on 6.3 million data collected from 14 diverse datasets and leverage them to predict the affine-invariant depth maps of consecutive RGB frames. Due to the use of the supervision of virtual normal in edge areas and planar regions of largescale and diverse datasets, the depth models can learn the geometric information well and be robust even in unseen scenarios.
However, the monocular depth model can only predict affine-invariant depth, which cannot be used for reconstruction directly due to the distortion and duplication caused by unknown scale and shift, as mentioned in Section 3.1 and Section 3.2. Traditional approaches recover a scale and a shift value for the depth map using least-squares fitting with some anchor points, which neglects the information of the distribution difference between affine-invariant prediction and anchor points. In contrast, we propose to perform locally weighted linear regression to recover the metric depth (Please see Section 4.2 in detail). Compared to the global least-squares fitting, our local recovery strategy generates a scale and a shift map for each depth map, which can not only recover metric depth, but also correct the overall depth maps and ensure the accuracy and consistency of 3D reconstruction. Now we can obtain the locally scale-aligned monocular video depth by simply per-frame depth prediction and local recovery, but the subtle details may still remain partially inconsistent between frames, which can bring outliers and duplication if we purely unproject it to 3D space without post-processing. Therefore, we propose to fuse multi-frame information with an RGB-D fusion module, which takes the RGB frames, depth maps, camera poses, and intrinsic parameters as input. It balances the difference between frames, filters out outliers and inconsistent regions between frames, and outputs the fused dense 3D mesh or point cloud.
In this work, we employ the TSDF fusion [64] to fuse multiple depth maps into a projective truncated signed distance function (TSDF) voxel volume during reconstruction. The sparse guided points used for local alignment can be obtained from various SLAM [65] systems, SfM [35], [59] algorithms, and some low-quality sensors such as ToF sensors of mobile phones. For the robust monocular depth estimation module, we train two strong and robust monocular depth estimation models, based on ResNet50 [33] and Swin-L [34] backbones, respectively. The image-level normalized regression loss [1], pair-wise normal loss [1] and multi-scale gradient loss [5] are employed during training for better learning the inherent geometric information of depth maps.
Our pipeline exploits the robustness of monocular depth estimation and the inherent consistency of sparse anchor points on consecutive frames, and handles some inconsistent regions with an RGB-D fusion module. Compared to current 3D reconstruction works, our pipeline is less likely to suffer from low-texture regions, occlusion, or lighting changes, which may lead to mismatch and inaccurate depth prediction, for MVS-based and unsupervised methods. Benefiting from the inherent consistency of sparse SfM points, our pipeline can further boost the performance and ensure better consistency of point cloud compared to consistent video depth estimation-based methods.

Metric Depth Recovery
Monocular depth estimation methods [1], [2], [3], [8] have achieved promising results on diverse scenes. The problem is that their predicted depth/inverse depth is scale-shift-invariant, namely, affine-invariant depth/inverse depth [6], [8]. Here we take the affine-invariant depth as an example. To recover the metric depth, it should be scaled and shifted, i.e., D * = sD + θJ , where D, s, θ and J are the predicted affine-invariant depth, scale, shift and all-ones matrix respectively. Some methods propose to obtain them through a global least-squares fitting method with groundtruth depth: where y is the flattened ground-truth metric depth, X is the homogeneous representation of the flattened predicted depth d, n = H ×W represents the flattened length of depth map with a shape of (H, W ). β is composed of scale value s and shift value θ, andβ is the optimized value of β. Note that the scale value s and shift value θ can be regarded as a scale map S and a shift map Θ shared on the whole map.
However, such a globally scaling and shifting method often fails to reduce the spatially heteroscedastic errors, which follow a rather simple pattern. For example, we visualize the pixel-wise absolute relative error map between the ground truth and the globally recovered predicted depth in Figure 5(a), and observe the existence of the low-frequency spatial error. We see that the left part has a higher error than that of the right part. Motivated by this observation, we propose to leverage a locally recovering method, i.e., locally weighted linear regression (LWLR), to recover a scale and a shift map. Guided by very sparse ground-truth points, we can fix and quantify these low-rank spatial errors which are common in depth estimation tasks. Locally Weighted Linear Regression. We thus employ a locally weighted regression method, which is: where y is the sampled sparse ground-truth metric depth (we use around 25∼100 points in practice), X is the homogeneous representation of the sampled sparse predicted depth d, m stands for the number of sampled points.
Different from recovering a scale-shift value or a globally shared scale-shift map by the global least-squares fitting method, we recover a location-aware scale-shift map. For each 2D coordinate (u, v), the predicted depth d can be fitted to the ground-truth depth y by minimizing the squared locally weighted 2 distance, which is re-weighted by a diagonal weight matrix W u,v . It pays more attention to sparse points closer to the estimated location, based on the idea that points near each other in the explanatory variable space are more likely to be related in a simple way. By iterating over the whole image, the scale map S and shift map Θ can be generated composed of the scale values s u,v and shift values θ u,v of each location (u, v). Finally, the locally recovered metric depth D * equals to the shift map Θ plus the Hadamard product ( , known as element-wise product) of the affine-invariant depth D and the scale map S. In our implementation, we employ a Gaussian kernel function to compute the weight matrix: where b is the bandwidth of Gaussian kernel, and dist i is the Euclidean distance between the guided point (u i , v i ) and target point (u, v). The scale map and shift map obtained this way can yield much more accurate metric depth than the global methods. However, some scale maps can be fitted to negative due to the shift-invariant characteristic of monocular depth and flexibility of weighted linear regression, which inverses the distribution of depth prediction and lacks reasonableness. The distribution of scale is shown in Figure 6 (a). Since the bias is not centered and the solution space is not bounded, results are widely distributed with no physical meanings and far from the real scale and shift. Therefore, we conduct the global recovery strategy to monocular depth first, and restrict the solution to be simple by adding an 2 regularization on the shift: where X is the homogeneous representation of the globally recovered depth d. With the regularization of shift, the location-related scale map is encouraged to be positive, as shown in Figure 6(b).  [67] 114K Outdoor High-quality Hypersim [68] 298K Indoor High-quality GraspNet [69] 82K Indoor High-quality Tartanair [70] 289K Synthetic High-quality UASOL [71] 162K Outdoor Mid-quality DrivingStereo [72] 182K Outdoor Mid-quality DIML [73] 121K Outdoor Mid-quality HoloPix50k [74] 99K Indoor Low-quality KeystoneDepth [75] 74K In the wild Low-quality WSVD [47] 117K In the wild Low-quality HRWSI [4] 18K Outdoor Low-quality MegaDepth [5] 79K Outdoor Low-quality DiverseDepth [6] 90K In the wild Low-quality Total 6.3M --With our proposed local recovery strategy, we only need very sparse ground-truth depth (around 25-100 points) to recover the metric depth map by fitting a location-related scale map and a shift map. Figure 5 compares the global least-squares fitting and our proposed weighted linear regression results. Thanks to the optimized pixel-wise scale map ( Figure 5(c)) and shift map ( Figure 5(d)), the overall loss is reduced considerably (Figure 5(b)). Note that the predicted affine-invariant depths are the same for the two methods. Importantly, the recovered metric depth with our method is more linearly correlated to the ground truth (see Figure 5(e) and Figure 5(f)). Please see Figure 12 for more examples.

EXPERIMENTS
Training Datasets. To train a robust monocular depth estimation model, 6.3 million diverse RGB-D pairs are processed and trained following the previous work LeReS [1], and the composition of datasets is listed in Table 2. Implementation Details. Following [1], [8], we balance high-quality, middle-quality, and low-quality data in each batch to ensure that they account for almost the same ratio in each batch. For example, each data accounts for N/3 if the batch size is set to N during training. Multiple data augmentation techniques are used, such as random flipping, color transformations, image blur, etc. We randomly crop the image by 448 × 448 for training. For our ResNet50 [33] model, the ImageNet pre-trained weight is adopted in our experiments. We use SGD with momentum, start training with a learning rate of 0.04 and decay it with 0.1 for 4 times. The batch size is set to 128. For our Swin-L model, we simply modify the large encoder of Swin Transformer [34] and follow the decoder of LeReS [1]. As for training, we strictly follow the process of Swin Transformer; i.e., it is trained with the optimizer of AdamW [66], an initial learning rate of 0.00006, a poly learning rate schedule and a batch size of 64. We employ our ResNet50 model for experiments of 3D scene reconstruction and ablation study. Evaluation Details. For monocular depth estimation, the absolute mean relative error (Rel) and the percentage of pixels with δ 1 = max d pred dgt , dgt d pred < 1.25 are evaluated on 5 zero-shot datasets including KITTI [76], NYU [18], ScanNet [43], ETH3D [77] and DIODE [44]. The rank means average ranking on the 5 aforementioned datasets. For 3D scene reconstruction, the evaluation experiments are performed on 5 NYU videos. Besides the Rel and δ 1 , the Chamfer l 1 distance and the F-score [78] are computed between point clouds of prediction and ground truth after TSDF fusion [64]. The threshold of the F-score is set to 5 cm.

Dense 3D Scene Reconstruction
With our model trained on 6.3 million data and the local scale and shift recovery method, we can achieve highquality 3D scene reconstruction through per-frame prediction and TSDF fusion [64]. To evaluate the consistency and accuracy, we collect 5 NYU videos and compare with the single image 3D reconstruction method (LeReS [1]), the state-of-the-art depth completion method (NLSPN [29]), the latest robust consistent video depth estimation method (RCVD [28]), the unsupervised video depth estimation method SC-DepthV2 [24], and the learning-based MVS method DPSNet [56]. Note that NLSPN and SC-DepthV2 are trained on NYU, and only NLSPN can predict metric depth. Our method uses the same sparse ground truth (100 points) as NLSPN. For LeReS and RCVD, we align their predictions with metric depth globally before evaluation. For SC-DepthV2 and DPSNet, only global scale values are recovered by ensuring the same medians as the ground truth. Besides leveraging sparse ground truth, we also sample points from SfM methods, e.g. COLMAP [59], to reconstruct a 3D scene with only an RGB video, ground-truth intrinsic and poses. The Rel, δ 1 , the Chamfer l 1 distance and the F-score with the threshold of 5 cm are employed for evaluation.
Quantitative comparisons are shown in Table 3. First, we compare with the depth completion method NLSPN [29], which also uses the sparse guided points to obtain metric information. The main difference is that their model should be trained on the testing set and lacks generalization in the wild. By contrast, we can achieve better performance and generalize well to zero-shot datasets due to the robust depth prior. RCVD [28] and SC-DepthV2 [24] aim to solve the visual consistency problem for video depth prediction. LeReS [1] reconstructs 3D scene shape from a single image and performs well in the wild. DPSNet [56] leverages CNNs to extract features and match between frames automatically. Compared with them, our method achieves state-of-the-art in accuracy and consistency.
Experiments of qualitative comparison are shown in Figure 7. Depth completion method NLSPN performs well overall but misses some high-quality details and shapes due to the lack of geometry supervision during training. The RCVD focuses on visual depth consistency but fails to recover the shift depth maps, leading to the distortion of the 3D structure, see the walls and outliers. The SC-depthV2 achieves visually consistent video depth through an unsupervised paradigm, but the weak supervision brings some distortion during reconstruction. The LeReS achieves excellent detail prediction but lacks consistency between frames for misalignment caused by the global recovery strategy. The DPSNet improves the quality of extracted features with the help of CNNs, but without training on the NYU dataset, it lacks robustness to generalize to some unseen scenarios. With our local recovery strategy, our method can reconstruct better 3D point clouds than others. For Ours-SfM (local), we obtain SfM depth first, then fit the monocular depth with SfM depth and filter out the errors bigger than the 99th-percentile iteratively, before performing the local recovery strategy. Note that even with slightly inaccurate sparse SfM points, Ours-SfM (local) can still achieve comparable results with Ours(global), which requires ground-truth depth acquired from sensors.

Monocular Depth Estimation
Comparison with State-of-the-Art Methods. In this experiment, we compare with state-of-the-art robust monocular depth estimation methods on five zero-shot datasets, whose scale and shift are recovered with a globally least-squares fitting method. During evaluation, the latest released model weights are adopted uniformly. As shown in Table 4, our Swin-L model achieves comparable results with DPT [2]. Through recovering scale and shift with the proposed locally weighted linear regression method, our method with ResNet50 and Swin-L backbones (i.e., "Ours-R50 (local)", "Ours-swin (local)") can outperform all previous methods and our global predictions by a large margin over all zeroshot testing datasets.

Effectiveness of Locally Weighted Linear Regression.
To demonstrate our proposed locally weighted linear regression can boost various monocular depth estimation methods, we enforce it on several different methods: 1) learning affine-invariant depth methods, e.g., LeReS [1], MiDaS [3], and DPT [2]; 2) learning metric depth on a specific dataset (VNL [8]); 3) learning scale-invariant depth with unsupervised methods (MonoDepth2 [25]); 4) depth completion method (NLSPN [29]). Results are shown in Table 5. We uniformly sample 100 guided points to perform the local recovery, and all their performances can be boosted significantly (see the "w" columns). Critically, even though the NLSPN method has input such 100 sampled points for completion, our method can still further boost its performance. Note that the latest released weights and code are used for this experiment. Decoupling of Monocular Depth Error. Besides improving performance, the local recovery strategy is also performed to decouple monocular depth error between ground truth and globally aligned prediction into coarse misalignment error and detail-missing error. Compared with the error of global recovery, the alleviated error brought by local recovery represents the coarse misalignment error, and the remaining one stands for detail-missing error. As shown in Table 5, the percentage of coarse misalignment error in red nearly remains consistent between affine-invariant depth estimation, which may reveal the drawback of coarse misalignment of monocular depth estimation. The bottleneck of NLSPN [29] and MonoDepth2 [25] is detail-missing for the alleviated error in percentage remains small.

Ablation Study
Ablation Study for Training Data. In this experiment, we aim to study the relations between data volume and performance improvement. We gradually aggregate more data for training, and evaluate the performance on 5 zeroshot datasets. Note that 3 different quality data sources Figure 7: Qualitative comparison of 3D scene reconstruction from video. We compare our algorithm with six representative methods: single image depth estimation method LeReS [1] aligned with ground-truth depth, robust depth estimation method RCVD [28] with iterations of consistency optimization between frames, depth completion method NLSPN [29] with sparse ground truth, unsupervised video depth estimation method SC-DepthV2 [24] with photometric and geometric consistency between frames, COLMAP [59], and learning-based MVS methods DPSNet [56] on ten NYU scenarios. Note that NLSPN and SC-DepthV2 have been trained on the NYU dataset before inferencing. The superscript stars * stands for the employment of ground-truth depth from sensors. Ours-SfM (local) means performing the local recovery strategy with sparse points sampled from SfM [35], [59] depth. As can be seen, our method reconstructs better 3D shapes in terms of both accuracy and consistency (see red arrows). Please zoom in for geometry details and shapes. 3: Quantitative comparison of monocular depth estimation and 3D scene reconstruction with single image depth estimation method LeReS [1], consistent video depth estimation method RCVD [28], unsupervised video depth estimation method SC-DepthV2 [24], depth completion method NLSPN [29], and learning-based MVS method DPSNet [56] on five NYU scenarios. Note that NLSPN and SC-depthV2 have been trained on the NYU dataset before evaluating. The "global" and "local" represent global and our proposed local metric depth recovery strategies separately. Ours-SfM (local) means performing the local recovery strategy with points sampled from SfM [35], [59] depth. As a result, our pipeline of 3D scene reconstruction from video achieves state-of-the-art performance on all the five scenes.
Method Type Affineinvariant 14.8 8.0 (−6.8) −46% 8.  Table 6. We can observe that when the data size increases from 42K to 900K (around by 20 times), the performance is boosted significantly. However, when further increased by 7 times, the accuracy can only be improved slightly. We conjecture that such large-scale data has fully exploited the capacity of the model (ResNet50 backbone).
Furthermore, we also conduct local recovery here to de-   couple the error into coarse misalignment error and detailmissing error. As shown in Table 6, the percentage of coarse misalignment error remains nearly constant, which shows the model can study the detail information and the global structure simultaneously. Figure 9 shows an overall decrease of mean average error and standard deviation for the scale and shift map. It represents an enhancement of the ability to generate accurate global structure without the help of local recovery.

Ablation Study for Locally Weighted Linear Regression
Method. The performance of our proposed local recovery strategy may be affected by the amount of sparse points, the sparsity distribution, random noises from sparse points, and the bandwidth b. Their effects on the depth accuracy are explored and shown in Table 7. All experiments are conducted on the NYU dataset. According to the experiments, simple 5 × 5 ground-truth depth can be leveraged to recover metric depth and improve 7: Ablation study for parameters of our proposed local recovery strategy. Here, "Amount", "Distribution", and "Noise" correspond to the number, distribution, and maximum perturbation percentage of the sampled ground truth. Parameter b represents the bandwidth of the Gaussian kernel function. The whole image and half image stand for only sampling ground truth from the original whole or half image.  accuracy. More ground truth leads to more performance boost, please see Table 8 and Figure 10 for more detailed analysis. Our strategy performs robust to the amount, distribution and noise of sparse points, but the overly concentrated sampling strategy should be avoided in practice. As for bandwidth b, it represents the effect of distance on the weight matrix. Experimentally, we suggest simply setting parameter bandwidth to the value l n , where l is the width of the RGB image and n is the number of sampled ground-truth points of one side. More precisely, if we sample 10 × 10 ground-truth points for a 500 × 500 image, the parameter bandwidth can be set to 50.
The effects of noises of sparse points to depth estimation and 3D reconstruction on three NYU [18] videos are quantitatively and qualitatively studied in Table 9 and Figure 11 separately. Our local recovery strategy can also perform well even with sparse points perturbed by random noise. Rel (%)

Ground-Truth Points
Local-Uniform Global-Uniform Local-Grid Global-Grid Figure 10: Comparison of local and global recovery strategies of predicted depth, uniform, and grid sampling strategies of ground-truth points during recovering monocular metric depth. With our proposed local recovery, the performance of monocular depth estimation improves significantly with the increase of ground-truth points. Besides, the grid sampling strategy outperforms the uniform one, which can reduce the requirements for ground-truth points. TABLE 9: Analysis for the effect of random noise in sparse points during reconstruction. To simulate the cases in practice, we bring random noise to all the ground-truth points, with the maximum random perturbation percentage ("Noise" in this table). Our local recovery strategy shows some robustness to the noise of sparse points. Here, we adopt ground-truth sparse points as "0%".

CONCLUSION
In this paper, we have leveraged the robust data-driven monocular depth estimation model, local recovery strategy and an RGB-D fusion module to implement a complete dense 3D scene reconstruction pipeline. Compared to existing 3D reconstruction methods, our pipeline achieves improved robustness, accuracy and consistency along consecutive RGB frames. Extensive experiments show that our method demonstrates a significantly better generalization  Figure 11: Illustration of the effect of the perturbation to depth estimation and 3D reconstruction. The performance decreases consistently but achieves acceptable results on three scenes with the increase of noise. Note that the threshold of 5 cm is adopted during calculating the F-score.  Figure 12: Analysis for depth error map, scale map, and prediction-GT curve of monocular depth estimation. Our proposed local recovery strategy can generate an oppositely distributed scale map compared with the error map of global recovery, which can alleviate the coarse misalignment of predicted depth and improve the linear relation of prediction-GT pairs. ability to monocular depth estimation and 3D scene reconstruction.
The proposed local recovery strategy can not only improve the accuracy and consistency of depth estimation significantly with robustness to both the amount and randomly-generated noises of the ground truth, but also can be an analytical tool to expose the shortcomings of existing depth estimation methods.