{'fraction_non_alphanumeric': 0.035959438377535104, 'fraction_numerical': 0.015366614664586583, 'mean_word_length': 4.78803611738149, 'pattern_counts': {'":': 4, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation

Self-supervised monocular depth estimation that does not require ground-truth for training has attracted attention in recent years. It is of high interest to design lightweight but effective models, so that they can be deployed on edge devices. Many existing architectures benefit from using heavier backbones at the expense of model sizes. In this paper we achieve comparable results with a lightweight architecture. Specifically, we investigate the efficient combination of CNNs and Transformers, and design a hybrid architecture Lite-Mono. A Consecutive Dilated Convolutions (CDC) module and a Local-Global Features Interaction (LGFI) module are proposed. The former is used to extract rich multi-scale local features, and the latter takes advantage of the self-attention mechanism to encode long-range global information into the features. Experiments demonstrate that our full model outperforms Monodepth2 by a large margin in accuracy, with about 80% fewer trainable parameters.


Introduction
Many applications in the field of robotics, autonomous driving, and augmented reality rely on depth maps, which represent the 3D geometry of a scene. Since depth sensors increase costs, research on inferring depth maps using Convolutional Neural Networks (CNNs) from images emerged. With the annotated depth one can train a regression CNN to predict the depth value of each pixel on a single image [10,11,22]. Lacking large-scale accurate dense ground-truth depth for supervised learning, self-supervised methods that seek supervisory signals from stereo-pairs of frames or monocular videos are favorable and have made great progress in recent years. These methods regard the depth estimation task as a novel view synthesis problem and minimize an image reconstruction loss [5,14,15,40,43]. The camera motion is known when using stereo-pairs of im-input Monodepth2 R-MSFM6 Lite-Mono (Ours) Figure 1. Our full model Lite-Mono has fewer parameters than Monodepth2 [15] and R-MSFM [44], but generates more accurate depth maps.
age, so a single depth estimation network is adopted to predict depth. But if only using monocular videos for training an additional pose network is needed to estimate the motion of the camera. Despite this, self-supervised methods that only require monocular videos are preferred, as collecting stereo data needs complicated configurations and data processing. Therefore, we also focus on training with monocular videos. In addition to increasing the accuracy of a monocular training by introducing improved loss functions [15] and semantic information [5,21] to mitigate the occlusion and moving objects problems, many work focused on designing more effective CNN architectures [17,32,38,40,44]. However, the convolution operation in CNNs has a local receptive field, which cannot capture the long-range global information. To achieve better results a CNN-based model can use a deeper backbone or a more complicated architecture [15,27,42], which also results in a larger model size. The recently introduced Vision Transformer (ViT) [8] is able to model global contexts, and some recent works apply it to monocular depth estimation architectures [3,34] to obtain better results. However, the expensive calculation of the Multi-Head Self-Attention (MHSA) module in a Transformer hinders the design of lightweight and fast inference models, compared with CNN models [34].
In this paper we pursue a lightweight and efficient selfsupervised monocular depth estimation model with a hybrid CNN and Transformer architecture. In each stage of the proposed encoder a Consecutive Dilated Convolutions (CDC) module is adopted to capture enhanced multi-scale local features. We then use a Local-Global Features Interaction (LGFI) module to calculate the MHSA and encode global contexts into the features. To reduce the computational complexity we calculate the cross-covariance attention [1] in channel dimension instead of the spatial dimension. The contributions of our method can be summarized in three aspects.
• We propose a new lightweight architecture, dubbed Lite-Mono, for self-supervised monocular depth estimation. We demonstrate its effectiveness with regard to the model size and FLOPs.
• The proposed architecture shows superior accuracy on the KITTI [13] dataset compared with competitive larger models. It achieves the state-of-the-art with the least trainable parameters. The model's generalization ability is further validated on the Make3D [31] dataset.
Additional ablation experiments are conducted to verify the effectiveness of different design choices.
• The inference time of the proposed method is tested on a Nvidia Titan XP and a Jetson Xavier platform, which demonstrates its good trade-off between model complexity and inference speed.
The remainder of the paper is organized as follows. Section 2 reviews some related research work. Section 3 illustrates the proposed method in details. Section 4 elaborates on the experimental results and discussion. Section 5 concludes the paper.

Monocular depth estimation using deep learning
Single image depth estimation is an ill-posed problem, because a 2D image may correspond to many 3D scenes at different scales. Methods using deep learning can be roughly divided into two categories. Supervised depth estimation. Using ground-truth depth maps as supervision, a supervised deep learning network is able to extract features from input images and learn the relationship between depth and RGB values. Eigen et al. [10] first used deep networks to estimate depth maps from single images. They designed a multi-scale network to combine global coarse depth maps and local fine depth maps. Subsequent works introduced some post-processing techniques, such as Conditional Random Fields (CRF), to improve the accuracy [23,24,37]. Laina et al. [22] proposed to use a new up-sampling module and the reverse Huber loss to improve the training. Fu et al. [11] adopted a multi-scale network, and treated the depth estimation as an ordinal regression task. Their method achieved higher accuracy and faster convergence.
Self-supervised depth estimation. Considering that large-scale annotated datasets are not always available, selfsupervised depth estimation methods that do not require ground-truth for training have attracted some attention. Garg et al. [12] regarded the depth estimation as a novel view synthesis problem, and proposed to minimize a photometric loss between an input left image and the synthesized right image. Their method was self-supervised, as the supervisory signal came from the input stereo pairs. Godard et al. [14] extended this work and achieved higher accuracy by introducing a left-right disparity consistency loss. Apart from using stereo pairs the supervisory signal can also come from monocular video frames. Zhou et al. [43] trained a separate multi-view pose network to estimate the pose between two sequential frames. To improve the robustness when dealing with occlusion and moving objects they also used an explainability prediction network to ignore target pixels that violate view synthesis assumptions. To model dynamic scenes other works introduced multitasks learning, such as optical flow estimation [40] and semantic segmentation [5,20], or introduced additional constraints, such as uncertainty estimation [29,39]. Godard et al. [15] found that without introducing extra learning tasks they could achieve competitive results by simply improving the loss functions. They proposed Monodepth2, which used a minimum reprojection loss to mitigate occlusion problems, and an auto-masking loss to filter out moving objects that have the same velocity as the camera. We also build our self-supervised training strategy based on their work.

Advanced architectures for depth estimation
Network architectures also play an important role in achieving good results in monocular depth estimation. By replacing the network architecture from the VGG model [32] with a ResNet [17] Yin et al. [40] achieved better results. Yan et al. [38] used a channel-wise attention module to capture long-range multi-level information and enhance local features. Zhou et al. [42] also used an attention module to obtain a better feature fusion. Zhao et al. [44] proposed a small architecture that used a feature modulation module to learn multi-scale features, and demonstrated the method's superiority. To reduce model parameters they only used the first three stages of ResNet18 [17] as the backbone. With the rise of vision transformer (ViT) [8] recent work applied it to various computer vision tasks [4,16,28,30,33] Figure 2. Overview of the proposed Lite-Mono. Our Lite-Mono has an encoder-decoder DepthNet for depth prediction, and a commonly used PoseNet [15,44] to estimate poses between adjacent monocular frames. The encoder of the DepthNet consists of four stages, and we propose to use Consecutive Dilated Convolutions (CDC) modules and Local-Global Features Interaction (LGFI) modules to extract rich hierarchical features. The details of these modules are shown in Figure 3.
al. [34] adopted the Dense Prediction Transformer [30] for self-supervised monocular depth estimation, and added another prediction head to estimate the camera's intrinsics. Bae et al. [3] proposed a hybrid architecture of CNN and Transformer that enhanced CNN features by Transformers. However, due to the high computational complexity of Multi-Head Self-Attention (MHSA) in a ViT, the abovementioned Transformer-based methods have more trainable parameters, and have a large speed gap compared with methods only using CNNs [34]. In this paper we want to fill this gap, and explore a lightweight and effective hybrid architecture design by taking advantages of both CNNs and Transformers.

Design motivation and choices
Several papers demonstrated that a good encoder can extract more effective features, thus improving the final result [15,17,42]. This paper focuses on designing a lightweight encoder that can encode effective features from the input images. Figure 2 shows the proposed architecture. It consists of an encoder-decoder DepthNet (Section 3.2) and a PoseNet (Section 3.3). The DepthNet estimates multi-scale inverse depth maps of the input image, and the PoseNet estimates the camera motion between two adjacent frames. Then, a reconstructed target image is generated, and the loss is computed to optimize the model (Section 3.4).
Enhanced local features. Using shallow instead of deeper networks can effectively reduce the size of a model. As mentioned shallow CNNs have very limited receptive fields, while using dilated convolution [41] is helpful to enlarge receptive fields. By stacking the proposed Consecutive Dilated Convolutions (CDC) the network is able to "observe" the input at a larger area, while not introducing extra training parameters.
Low-computation global information. The enhanced local features are not enough to learn a global representation of the input without the help of Transformers to model long-range information. The MHSA module in the original Transformer [8] has a linear computational complexity to the input dimension, hence it lim-its the design of lightweight models. Instead of computing the attention across the spatial dimension the proposed Local-Global Features Interaction (LGFI) module adopts the cross-covariance attention [1] to compute the attention along the feature channels. Comparing with the original self-attention [8] it reduces the memory complexity from where h is the number of attention heads. In the following we introduce the proposed architecture in detail.

DepthNet
Depth encoder. The proposed Lite-Mono aggregates multi-scales features across four stages. The input image with size H × W × 3 is first fed into a convolution stem, where the image is down-sampled by a 3 × 3 convolution. Following two additional 3 × 3 convolutions with stride = 1 for local feature extraction, we get the feature maps with size H 2 × W 2 ×C 1 . In the second stage the features are concatenated with the pooled three-channel input image, and another 3 × 3 convolution with stride = 2 is adopted to down-sample the feature maps, resulting in feature maps with size H 4 × W 4 × C 2 . Concatenating features with the average-pooled input image in a down-sampling layer can reduce the spatial information loss caused by the reduction of feature size, which is inspired by ESPNetv2 [3]. We then use the proposed Consecutive Dilated Convolutions (CDC) module and the Local-Global Features Interaction (LGFI) module to learn rich hierarchical feature representations. The down-sampling layers in the second and the third stage also receive the concatenated features output by the previous down-sampling layer. This design is similar to the residual connection proposed in ResNet [17], and is able to model better cross-stage correlation. Similarly, the output feature maps are further fed into the third and the fourth stages, and output features of dimension The proposed CDC module utilizes dilated convolutions to extract multi-scale local features. Different from using a parallel dilated convolution module only in the last layer of the network [6] we insert several consecutive dilated convolutions with different dilation rates into each stage for adequate multi-scale contexts aggregation.
Given a two-dimensional signal x[i] we can define the output y[i] of a 2D dilated convolution as: where w[k] is a filter with length K, and r denotes the dilation rate used to convolve the input x[i]. In a standard non-dilated convolution r = 1. By using a dilated convolution we can keep the size of the output feature map fixed while achieving a larger receptive field. Considering an input feature X with dimension H ×W ×C our CDC module outputsX as follows: where Linear G denotes a point-wise convolution operation, followed by the GELU [18] activation. BN is a batch normalization layer, and DDW Conv r (·) is a 3 × 3 depthwise dilated convolution with dilation rate r.
Local-Global Features Interaction (LGFI). Given an input feature map X with dimension H × W × C we linearly project it to the same dimensional queries Q = XW q , keys K = XW k , and values V = XW v , where W q , W k , and W v are weight matrices. We use the cross-covariance attention [1] to enhance the input X: where Attention(Q, K, V ) = V · Sof tmax(K T · Q). Then, we increase the non-linearity of the features: where LN is a layer normalization [2] operation. According to different channel numbers, CDC blocks and dilation rates, we design three variants of the depth encoder. Table 1 shows more details. Depth decoder. Different from using complicated upsampling method [44] or introducing additional attention modules [3] we use a depth decoder adapted from [15]. As shown in Figure 2 it increases the spatial dimension using bi-linear up-sampling, and uses convolutional layers to concatenate features from three stages of the encoder. Each up-sampling block follows a prediction head to output the inverse depth map at full, 1 2 , and 1 4 resolution, respectively.

PoseNet
Following [15,44] this paper uses the same PoseNet for pose estimation. To be specific, a pre-trained ResNet18 is used as the pose encoder, and it receives a pair of color images as input. A pose decoder with four convolutional layers is used to estimate the corresponding 6-DoF relative pose between adjacent images.

Self-supervised learning
Different from the supervised training that utilizes ground-truth of depth we treat the depth estimation as the task of image reconstruction. Similar with [43] the learning objective is modeled to minimize an image reconstruction loss L r between a target image I t and a synthesized target imageÎ t , and an edge-aware smoothness loss L smooth constrained on the predicted depth map D t .
Image reconstruction loss. The photometric reprojection loss is defined as: whereÎ t can be obtained by a function F in terms of the source image I s , the estimated pose P , the predicted depth D t , and the camera's intrinsics K. As introduced in [43] L p is computed by a sum of the pixel-wise similarity SSIM (Structural Similarity Index [36]) and the L1 loss between I t and I t : where α is set to 0.85 empirically [15]. In addition, to deal with out-of-view pixels and occluded objects in a source image the minimum photometric loss [15] is computed: where I s can be either the previous or the next frame with respect to the target image. Another binary mask [15] is used to remove moving pixels: Therefore, the image reconstruction loss is defined as: Edge-aware smoothness loss. To smooth the generated inverse depth maps an edge-aware smoothness loss is calculated, followed by [15,42]: where d * t = d t /d t denotes the mean-normalized inverse depth. The total loss can be expressed as: where s is the different scale output by the depth decoder. We set λ = 1e −3 as in [15].

Experiments
In this section we evaluate the proposed framework, demonstrating the superiority of our method.

Datasets
KITTI. The KITTI [13] dataset contains 61 stereo road scenes for research in autonomous driving and robotics, and it was collected by multiple sensors, including camera, 3D  Table 2. Comparison of our models to some recent representative methods on KITTI benchmark using the Eigen split [9]. All input images are resized to 640 × 192 unless otherwise specified. The best and the second best results are highlighted in bold and underlined, respectively. "M": KITTI monocular videos, "M+Se": monocular videos + semantic segmentation, "*": input resolution 1024 × 320, "M †": without pre-training on ImageNet [7].  Lidar, GPU/IMU, etc. To train and evaluate the proposed method we use the data split of Eigen et al. [9], which has a total of 39,180 monocular triplets for training, 4,424 for evaluation, and 697 for testing. The self-supervised training is based on the known camera intrinsics K, as indicated in Eq. 5. By averaging all the focal lengths of images across the KITTI dataset we use the same intrinsics for all images during training [15]. In the evaluation the predicted depth is restricted in the range of [0, 80]m, as is common practice.
Make3D. To evaluate the generalization ability of the proposed method it is further tested on the Make3D [31] dataset, which contains 134 test images of outdoor scenes. We load the model trained on the KITTI dataset, and directly infer it on these test images.

Implementation details
Hyperparameters. The proposed method is implemented in PyTorch, and it is trained on a single Nvidia Titan Xp with the batch size of 12. We use AdamW [26] as the optimizer, and set the weight decay to 1e −2 . For models trained from scratch an initial learning rate of 5e −4 with a cosine learning rate schedule [25] is adopted, and the training epoch is set to 35. It is found that pre-training on Ima-geNet [7] make the network converge fast, so the network is trained for 25 epochs when using pre-trained weights, and the initial learning rate is set to 1e −4 . A monocular training for 35 epochs takes about 15 hours.
Data augmentation. Data augmentation is adopted as a preprocessing step to improve the robustness of the training. To be specific, the following augmentations are performed with a 50% chance: horizontal flips, brightness adjustment (±0.2), saturation adjustment (±0.2), contrast adjustment (±0.2), and hue jitter (±0.1). These adjustments are applied in a random order, and the same augmentation method is also used by [15,27,44].

KITTI results
The proposed framework is compared with other representative methods with model sizes less than 35M, and the results are shown in Table 2. The full model Lite-Mono achieves the best results, while the model size is the smallest (3.1M). Specifically, Lite-Mono greatly exceeds Mon-odepth2 [15] with a ResNet18 [17] backbone, but the model size is only about one-fifth of this model. It also outperforms the ResNet50 version of Monodepth2, which is the largest model (32.5M) in this table. Besides, the proposed model beats the recent well-designed small model R-MSFM [44]. Comparing with the new MonoFormer [3] with a ResNet50 backbone the proposed Lite-Mono out-performs it in all metrics. Our other two smaller models also achieve satisfactory results, considering that they have fewer trainable parameters. Figure 4 shows that the proposed Lite-Mono achieves satisfactory results, even on challenging images where moving objects are close to the camera (column 1).

Make3D results
The proposed method is evaluated on the Make3D dataset to show its generalization ability in different outdoor scenes. The model trained on KITTI is directly inferred without any fine-tuning. Table 3 shows the comparison of Lite-Mono with other three methods, and Lite-Mono performs the best. Figure 5 shows some qualitative results. Owing to the proposed feature extraction modules Lite-Mono is able to model both local and global contexts, and perceives objects with different sizes.

Complexity and speed evaluation
The proposed models' parameters, FLOPs (floating point of operations), and inference time are evaluated on a Nvidia Titan XP and a Jetson Xavier, and are compared with Mon-odepth2 [15] and R-MSFM [44]. Table 4 shows that the proposed design has a good balance between model size and speed. Notice that our Lite-Mono-tiny outperforms Mon-odepth2 both in speed and accuracy (Table 2). Although R-MSFM [44] is a lightweight model, it is the slowest. Our models also infer fast on the Jetson Xavier, which makes it possible to deploy them on edge devices.  Table 5. Ablation study on model architectures. All the models are trained and tested on KITTI with the input size 640 × 192.

Ablation study on model architectures
To further demonstrate the effectiveness of the proposed model the ablation study is conducted to evaluate the importance of different designs in the architecture. We remove or adjust some modules in the network, and report their results on KITTI, as shown in Table 5.
The benefit of LGFI blocks. When all the LGFI blocks in stage 2, 3, and 4 are removed, the model size decreases by 0.4M, but the accuracy also drops. The proposed LGFI are crucial to make Mono-Lite encode long-range global contexts, thus making up for the drawback that CNNs can only extract local features.
The benefit of dilated convolutions. If all the dilation rates of convolutions in CDC blocks are set to 1, i.e., there is no dilated convolutions used in the network. It can be observed that although the model size remains the same the accuracy drops more than for not using LGFI blocks. The benefit of introducing the CDC module is to enhance the locality by gradually extracting multi-scale features, while not adding additional trainable parameters.
The benefit of pooled concatenations. When three pooled concatenations are removed the accuracy also decreases. This is because when using a down-sampling layer to reduce the size of feature maps, some spatial information is also lost. The advantage of using pooled concatenations is that the spatial information is kept, and this design only adds a small number of parameters (0.007M).
The benefit of cross-stage connections. When two cross-stage connections are removed the accuracy decreases slightly. The benefit of the proposed cross-stage connections in Mono-Lite is to promote feature propagation and cross-stage information fusion.

Ablation study on dilation rates
We study the influence of the dilation rate in the proposed CDC module on accuracy. Four different settings are used in this experiment. (1) The default setting of our models is to group every three CDC blocks together, and set the dilation rate to 1, 2 and 3, respectively. For the last three blocks we set them to 2, 4, and 6. (2) Based on the default setting we also set the dilation rates of the last three blocks to 1, 2, and 3, respectively. (3) Similar to the default setting we set 1, 2, and 5 in every three CDC blocks as a group. (4) Based on the default setting the dilation rates in the last two CDC blocks are set to 2, 4, 6 and 4, 8, 12, respectively. Table 6 lists the accuracy under different dilation settings. Comparing (2) with (3) the accuracy benefits from larger dilation rates. However, (4) using very large dilation rates in the late CDC blocks does not help. Simply pursuing larger dilation rates will result in the loss of local information, which is not good for the network to perceive small and medium-sized objects. Therefore, the proposed Lite-Mono adopts the setting (1) to extract multi-scale local features, i.e., smaller dilation rates are used in shallow layers, and they are doubled in the last three CDC modules. NO Table 6. Ablation study on dilation rates.

Conclusions
In this paper we present a novel architecture Lite-Mono for lightweight self-supervised monocular depth estimation. A hybrid CNN and Transformer architecture is designed to model both multi-scale enhanced local features and longrange global contexts. The experimental results on the KITTI dataset demonstrate the superiority of our method. By setting optimized dilation rates in the proposed CDC blocks and inserting the LGFI modules to obtain the localglobal feature correlations, Lite-Mono can perceive different scales of objects, even challenging moving objects closed to the camera. We also validate the model's generalization ability on the Make3D dataset. Besides, Lite-Mono achieves a good trade-off between model complexity and inference speed.