{'fraction_non_alphanumeric': 0.026047654918130277, 'fraction_numerical': 0.012012845418863735, 'mean_word_length': 5.097171863669326, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Pyramid Frequency Network with Spatial Attention Residual Refinement Module for Monocular Depth Estimation

Deep-learning-based approaches to depth estimation are rapidly advancing, offering superior performance over existing methods. To estimate the depth in real-world scenarios, depth estimation models require the robustness of various noise environments. In this work, a Pyramid Frequency Network(PFN) with Spatial Attention Residual Refinement Module(SARRM) is proposed to deal with the weak robustness of existing deep-learning methods. To reconstruct depth maps with accurate details, the SARRM constructs a residual fusion method with an attention mechanism to refine the blur depth. The frequency division strategy is designed, and the frequency pyramid network is developed to extract features from multiple frequency bands. With the frequency strategy, PFN achieves better visual accuracy than state-of-the-art methods in both indoor and outdoor scenes on Make3D, KITTI depth, and NYUv2 datasets. Additional experiments on the noisy NYUv2 dataset demonstrate that PFN is more reliable than existing deep-learning methods in high-noise scenes.


Introduction
Although Deep Convolutional Neural Networks(DCNNs) have attracted great attention in recent decades and have shown significant progress in computer vision enabled by the availability of large-scale labelled datasets, these models lack the robustness of the human vision system. It has been observed that numerous image corruptions, such as Gaussian noise, fog, overexposure, and blurring, can lead to tremendous degradation. 1 Improving network robustness is an important step towards safely deploying models in real-world scenarios.
Recently, the robustness of monocular depth estimation DCNNs has two common solutions, namely data augmentation and model redundancy. Data augmentation, a technique used to increase the amount of data by adding slightly modified copies of existing data, is a widely-used and constantly applicable approach to train robust models. Models apply image transformations to training data, such as flipping, cropping, adding random noise and adding occlusions. 2 Data augmentation acts as a regularizer and helps reduce over-fitting when training DCNNs. However, it is observed that data augmentation exacerbates the data imbalance and increases model training costs. Simple data augmentation methods have inevitable limitations due to the overemphasizing features of homogeneous instances. Model redundancy is another solution to enhance the robustness of model-based approaches. 3 Though the model redundancy method increases model size, it allows the model to choose reliable features. However, due to the notorious unreliability of DCNNs, model redundancy is an attractive option.
In this work, we propose a Pyramid Frequency Network(PFN) with Spatial Attention Residual Refinement Module(SARRM) for monocular depth estimation, which aims at handling the weak robustness of previous deep-learning methods. Specifically, the key contributions of the PFN can be summarized as follows: • Our approach is inspired by the Fourier perspective on model robustness, 4 and relies on frequency division and a feature pyramid network to extract frequency features. The frequency pyramid is proposed to separate the image into different frequencies, which alleviates the periodic noise issue and guides the model to reconstruct accurate depths..
• A spatial attention residual refinement module(SARRM) is presented for the information fusion between high-frequency and low-frequency domains. As an auxiliary module, it not only helps extract features from the RGB domain, but also necessitates the combination of preliminary depth and detailed information recovered from multi-stage frequency bands.
Thus, it can improve the feature extraction capability of the model, and is also useful in cases where only high-noise or severe-corruption scene images are available.
• We present experiments on three benchmark datasets, demonstrating that our PFN signifi-cantly improves the performance of monocular depth estimation. In particular, on the KITTI depth dataset, 5 the PFN achieves a new state-of-the-art accuracy of 95.3% when the threshold is 1.25, whereas the previous best result was 95.1%. In addition, extensive experiments show that our PFN achieves better visual accuracy than most existing methods under noisy circumstances.

Related work
In the past few years, the DCNNs have been applied to many fields in computer vision with great success. We briefly survey techniques related to our work, namely model-based depth estimation.
Most deep-learning-based methods are fully supervised, requiring ground-truth depth during training. Depth estimation is fundamental for understanding the stereoscopic of scenes from 2D images. From the beginning of depth estimation research, the majority of work has concentrated on geometry-based algorithms 67 that rely on point correspondences between images and triangulation.
A current popular approach is to construct a depth map using learning non-linear multi-channel to single-channel mapping, implemented as a CNN work. 8 Because the handcrafted features can only capture limited local information, the global relationship was incorporated using deep CNN and Markov Random Fields features 9 . 10 Another novel approach to using global information was the Depth-transfer, 11 which generates plausible depth maps from videos using non-parametric depth sampling. Eigen 12 employed two deep network stacks to incorporate global and local cues: one made a coarse global prediction based on the entire image, while the other refined this prediction locally. A single multi-scale convolutional network architecture 13 was applied for three independent computer vision tasks, namely depth prediction, normal surface estimation, and semantic labeling.
Eigen 13 proposed another method that captured more image details, and refined predictions using a sequence of scales.
Furthermore, a deep convolutional neural field model 14 17 also involved ordinal regression in the task of depth prediction. Xian 16 introduced a method to automatically generate dense relative depth annotations from web stereo images and proposed a ranking loss to deal with imbalanced ordinal relations, which enforces the network to focus on a set of hard pairs. Then, Xian 17 proved that the pair-wise ranking loss, combined with the structure-guided sampling strategies, benefits the depth prediction.
To address the issue of occluding contours, Ramamonjisoa introduced SharpNet, 18 a method that predicts an accurate depth map given a single input color image, with particular attention to the reconstruction of occluding contours. To facilitate the guidance of densely encoded features to the desired depth prediction, Lee 19 proposed a multi-scale local planar guidance technique for monocular depth estimation called Big to Small(BTS). This technique utilized novel local planar guidance layers located at multiple stages during the decoding phase.
Recently, Liu 20 proposed learning the affinity through a deep CNN with Spatial Propagation Networks, yielding better results comparing to the traditional manually designed affinity. However, depth refinement commonly necessitated a local affinity matrix rather than a global affinity matrix.
To fuse the discrete ground-truth depth points measured by LIDAR sensors, Cheng 21 presented the Convolutional Spatial Propagation Network(CSPN), in which the depths of all pixels are updated simultaneously within a local affinity. For fast and accurate monocular depth estimation, Lu 22 proposed an effective generative adversarial network. The approach demonstrated the feasibility of applying a dense-connected UNet for reducing information transmission loss and then finetuning the blur depth using the high-order Convolutional Spatial Propagation Network(GA-CSPN) with a modified discriminator loss function. Additionally, the loss function of the discriminator was modified by adding the correlation loss, which is used to measure the similarity of real and fake labels.
Besides fully-supervised depth estimation, another alternative is to train depth estimation models using a corresponding view image as supervision. Godard et al. 23 constructed Monodepth, an unsupervised Monocular Depth Estimation with Left-Right Consistency. Monodepth built a deep network to predict stereo pairs and predict their pixel disparities. Furthermore, Godard et al. 24 proposed the Monodepth2, a fully-resolution multi-scale sampling method that minimizes visual artifacts, and an auto-masking loss to ignore training pixels that violate camera motion assumptions.
For model-based depth estimation, the pyramid structure provided an efficient architecture to extract different types of information from various layers. Liao 25 assumed that most of the depth hypotheses up-sampled from a well-estimated depth map were accurate. Based on this assumption, Liao introduced a pyramid multi-view stereo network based on adaptive depth estimation, which gradually refines and up-samples the depth map to the desired resolution. To enhance the robustness in ill-posed regions or new scenes, Tian 26 came up with a self-improving pyramid stereo network that can not obtain a direct regression disparity without complicated post-processing.
Lee 27 proposed a single-image depth estimation method based on the Fourier domain analysis(FDA). The first step is generating multiple depth map candidates by cropping images with various ratios and then combining the multiple candidates in the frequency domain.

Preliminaries
Although great progresses has been made in the field of monocular depth estimation, estimating an accurate dense depth from a single image remains challenging, even for humans. One of the reasons is that it is an intractable problem due to the complexity and high noise of application scenarios in the real world. Therefore, to estimate the depth of real-world scenarios, these models require the robustness of various high-noise environments. For example, it has been observed that commonly occurring image corruptions, such as Gaussian noise, fog, overexposure, contrast change, and blurring, can lead to significant performance degradation. 28 Inspired by the Fourier perspective on model robustness, 4 we apply Fourier domain processing for depth estimation tasks. In contrast to the RGB domain, the Fourier domain makes it possible for the separation of most noises from corrupted images. However, Fourier features, which are complex frequency domain features, are notoriously difficult to apply directly to feature fusion tasks.
In the depth estimation task, frequency division is a feasible option that divides the Fourier features into multi-stage frequency bands that each contain different components of depth information.
To obtain the frequency response, each depth map is estimated from source images that have been divided into multiple frequency bands. The pipeline involves first separating RGB images into each frequency band that needs to be evaluated for the percentage of valid information. As  The absolute relative errors of the estimated depths obtained from multiple frequency bands are shown in Fig.2. It can be observed from Fig.2 that the basic U-net backbone 29 can achieve high accuracy, which is evaluated by the percentage of depth where the ratio of estimated and ground-truth depth is less than a threshold δ, using information from the input that would be unrecognizable to humans. From another perspective, the accuracy curve represents the percentage of valid information. However, the image pass through HPF loses most of valuable depth information and the most of depth information exists in the low-frequency domain. The valid information percentage of each frequency band is proportional to the absolute value of the fitted curve gradient. Therefore, the crucial information for the depth estimation task is mainly preserved in the frequency band between 5Hz and 30Hz.

Method
In order to fuse the Fourier information from multiple frequency bands, we construct a fully- Through frequency division, the information from multi-stage frequency bands can be obtained, thereby increasing the details of the reconstruction depth and improving the accuracy.
The backbone network, shared by the preliminary depth estimation modules, extracts features from each frequency pyramid stage. In our experiments, we adopt a High-Resolution network(HRnet) 30 as the preliminary depth estimation backbone. In the high-frequency module, each HPF pyramid layer, followed by a convolution layer, refines the blurred depth map with high-level information such as texture and edge features. The SARRM module, which combines the residual network with the spatial attention channel, helps to fuse the blur depth and single-layer features.

Pyramid frequency network
The 2D image frequency division is implemented by the 2D Fast Fourier Transform(2D-FFT) method which is an effective implement of 2D Fourier Transform(2D-FT). Frequency division can help to fuse the network features from multi-stage frequency bands, which is beneficial for the positive periodic signal. For the negative periodic signal, such as noise, the network tends to ignore the irrelevant information during the feature fusion. The bandwidth and cut-off frequencies in the frequency domain division method will be determined in ablation experiments.

End-to-end backbone
Numerous existing end-to-end networks are available for the PFN backbone, such as U-net, 29 ResNet, 31 Hourglass network 32 and deep High-Resolution network(HR-net). 30 In previous works, most end-to-end networks are directly transfer true-colour image to depth map. The backbone of PFN demonstrates the preliminary depth estimation, which requires further refinement in the following module. In order to employ a more effective and deeper network, HR-net 30 is chosen to be the backbone and shown as Fig. 4.

Spatial Attention Residual Refinement Module
The key idea of the refinement module is to define the intuitive combination of blur depth and the features from different frequency bands. Unlike the existing fusion methods, we place lo-  In contrast to the single-layer convolution module, SARRM can be used as multiple skip connections inside the decoding phase, allowing complicated relations between periodic features and the refined prediction. The spatial attention mechanism is dedicated to extracting the intrinsic re-lationships between spatial features. In terms of network complexity, the SARRM is acceptable in various application scenarios.

Loss function
We employ pixel-wise L2 loss, Mean Square Error(MSE), to measure the estimation error and then combine it with L1 loss, Mean Average Error(MAE), to fine-tune the PFN. The mix loss function is thus shown as follows: where y represents the ground-truth depth,ŷ represents the predicted depth and n represents its number of data pairs in the dataset.
In the mixed loss function, the L2 loss, which has a higher order than L1 loss, guides the optimization process in a more effective gradient descent direction. Because the derivative of the first-order function includes a constant term, the L1 loss is regarded as an optimization-related regularization term.

Experiment
In this section, we first describe the implementation details, the datasets information and benchmark performance of experiments. The PFN outperforms other exist monocular depth estimation methods and established a way to combine frequency components.
The learning rate of the model optimizer is set to 10 −4 initially and decreases to half every 100 epochs. In the training parameter set, the batch of data is set to 1. The PyTorch implement of PFN is trained with one RTX 2080 GPU, and this project is proposed online.
The implementation of 2D-FFT is accelerated by the PyTorch-fft package, which can be directly implemented by pixel vectorization in PyTorch.

Datasets
To evaluate the depth estimation task, we apply three benchmark datasets, Make3D, KITTI depth and NYUv2 datasets. The details are elaborated below.

Make3D dataset
The Make3D dataset 934 contains 534 outdoor images, 400 images for training and 134 images for testing. The dataset has images with a resolution of 2272×1704 and depth maps with 55×305.
Therefore, we resize the resolution of all the images to 512×512 and depth maps to 512×512.
Same as the pre-processing of previous works, the depth map C1, which range from 0m to 80m, and C2, which range from 0m to 70m, is reported by three commonly used evaluation metrics. 11

KITTI depth dataset
The KITTI depth dataset 5

Evaluation method
Following previous works, the depth map is evaluated by absolute relative error(REL), root mean square error(RMS), root mean square logarithmic error(RMS log ) 11 and three accuracy of different thresholds (1.25, 1.25 2 , 1.25 3 ). The specific formulas of these evaluation methods are shown below.
• Absolute relative error REL = 1 n y pred − y gt y gt y pred is depth estimated by network and y gt is the ground truth.
• Root mean square error • Root mean square logarithmic error RM S log = 1 n (log(y pred ) − log(y gt )) 2 • Root mean square logarithmic error RM S log = 1 n (log(y pred ) − log(y gt )) 2 (5) • Accuracy Percentage of depth y where the ratio of estimated and ground truth depth is less than a threshold.

Implementation
We first determine the cut-off frequencies of PFN through reasonable ablation experiments. Then we test DepthTransfer, 11 LRC-Deep3D, 36 Monodepth, 23  NYUv2 datasets. 35 All our experiments were run on the same machine with 32GB of RAM and one RTX 2080Ti GPU. Each experiment was conducted with an independent model during the training phase.

Ablation experiments
In order to select the appropriate cut-off frequency, we compare the accuracy and speed of PFN the with multiple cut-off frequencies over the NYUv2 dataset.
The cut-off frequency selection is a significant contributory factor to the improvement of PFN.
The experiments that follow aim to divide the multi-stage information in an appropriate method, thereby allocating valid information reasonably in frequency domains and identifying potential independent information that needs to be interacted with. In other word, this method extracts features independently from multi-stage frequency bands and avoids the homogeneity of data when the whole image is extracted.   Table.1. The experiment result in Fig.2 shows that most depth information is preserved in the frequency band between 5Hz and 30Hz. Thus, in the ablation experiment, dense cut-off frequencies are applied in the low-frequency part, while sparse cut-off frequencies are used in the highfrequency part. With the multi-layer modeling, the minimum REL of PFN is achieved at 0.1000, and the RMS is achieved at 0.0460.
Results on stronger backbones, by replacing U-net with HR-net, 30 are presented in Table.2. As seen in Table.2, the complex network structure of PFN results in the whole algorithm failing to operate in real-time on various sizes of input images. The increasing number of layers is found to be particularly effective in improving accuracy, and the five-layer pyramid network maintains a trade-off between accuracy and running time. Therefore, the cut-off frequencies of PFN are set as 5Hz, 10Hz, 50Hz and 100Hz. In this section, we compare the complexity and the accuracy between state-of-the-art strategies quantitatively. All the learning-based feature extraction methods systematically give better results than traditional methods.  As quantitatively shown in Table.5, the proposed end-to-end approach significantly outperforms both the existing fully-supervised and semi-supervised methods on the KITTI depth dataset.
On this dataset, the proposed method results in lower errors and higher accuracy compared to the existing fully-supervised methods. Without using extra data argumentation methods, the proposed method achieves 0.0690 REL on test, which is over 4% relative improvement over the previous best method, 22 and is also better than the concurrent work of Monodepth2's 0.1060. 24 With the proposed frequency-based method, the minimum REL of PFN is achieved at 0.1480, and the RMS is achieved at 3.8180 in C1 metric.  Fig.8. Without any tricks for data argumentation on the NYUv2 dataset,PFN gives a marginal improvement of 1.85% RMS over the previous best method, AdaBins, 50 on the test set. In the NYUv2 dataset, the proposed method has reached the best in six evaluation methods.
Conclusively, the proposed PFN outperforms all existing methods in all evaluation methods on the NYUv2 dataset.
Above experiments demonstrate that the five-layer PFN yields better performance than other existing model-based methods. The improvement of accuracy is expected because of the increased capacity of the network. However, it also results in longer training times. Hence, we resort to using the five-layer pyramid as the standard architecture in all our experiments. These experiments confirmed that the PFN is better than the state-of-the-art for most metrics on Make3D 9 , 34 KITTI depth 5 and NYUv2 datasets 35 datasets. Results of depth estimation on the Make3D 34 dataset are shown in Fig.6. Compared with NYUv2 and KITTI depth dataset, Make3D contains a relatively small number of samples. Therefore, the inferior visual performance of most deep-learning methods on Make3D due to over-fitting, such as the mismatch of colour and object. While CSPN and DORN mostly restore the depth from input, many of their results are shown as blurry and do not maintain the degree of sharpness as seen in the ground truth. Depth-Transfer even fails to reconstruct the proper edge between the object and background in the image by the style-transfer method. The PFN clearly reconstructs the most detailed depth map while properly smoothing the low-resolution ground truth. Reconstruction cases and their corresponding zoomed details of depth estimation on the KITTI depth dataset are shown in Fig.7. In zoomed depth maps, the depth map of PFN shows much more precise object boundaries. The car shown in the zoomed rectangle is reconstructed accurately by the proposed PFN algorithm compared to the other algorithms. In addition, PFN has a good visual performance for the zoomed car, but shows a slight visible ringing artifact on object boundaries. It can also be observed that there is a more accurate pedestrian reconstruction in the lane frame and a better reconstruction of the bush in the garden.

Robustness analysis
The previous section demonstrated the performance of existing model-based methods on widelyused datasets. In this section, we evaluate the performance of PFN on real-world indoor datasets with noisy environments. Specifically, we chose Gaussian noises for experiments. Same as previous experiments, images are resized into 512×512 and depth maps are resized into 512×512 in robustness experiments.

Gaussian noise
Gaussian noise is statistical noise having a probability density function equal to that of the normal distribution, which is also known as the Gaussian distribution. The probability density function p of a Gaussian random variable z is given by: where z represents the pixel value, µ represents the mean value and σ represents its standard deviation.
For pre-processing, the original images are added with random Gaussian noise and images under different noise variances are shown in Fig.9. σ is the variance of Gaussian noise. From the Fig.9, the slight noise, whose variance ≤ 10 −3 , are unperceivable for human vision.

Robustness experiment
To validate the robustness of existing deep-learning methods, we evaluate the performance of CSPN, 21 GA-CSPN 22 and PFN on the Gaussian noise additive dataset which is modified from the NYUv2 dataset.
As shown in Fig.10, the NYUv2 dataset with a gradual increase of Gaussian noise is evaluated As shown above, the proposed method is a robust end-to-end depth estimation approach that surpasses traditional fully-supervised methods. Because the pyramid frequency module can separate most aperiodic features from input images, the depth prediction is capable of extracting more periodic and valid features. Although the accuracy of the PFN decreases in the noisy dataset, it has achieved significant progress compared to previous methods.

Conclusion
We have presented a Pyramid Frequency Network(PFN), a frequency-based network to extract features from multiple frequency bands and find potential independent relations that need to be interacted with. To reconstruct depth maps with more accurate details, a Spatial Attention Residual Refinement Module(SARRM) is proposed to refine the blur depth. The proposed PFN is tested on three benchmark datasets, Make3D, KITTI depth and NYUv2 datasets and outperforms the stateof-the-art methods. With extra robustness testing on the noisy NYUv2 dataset, we have confirmed that PFN for monocular depth estimation in common scenes is more reliable than depth estimations obtained with state-of-the-art deep-learning methods. Comparison results on NYUv2 dataset between state-of-the-art strategies