{'fraction_non_alphanumeric': 0.028757494314657846, 'fraction_numerical': 0.007814761215629523, 'mean_word_length': 4.932180524895757, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DUT: Learning Video Stabilization by Simply Watching Unstable Videos

Previous deep learning-based video stabilizers require a large scale of paired unstable and stable videos for training, which are difficult to collect. Traditional trajectory-based stabilizers, on the other hand, divide the task into several sub-tasks and tackle them subsequently, which are fragile in textureless and occluded regions regarding the usage of hand-crafted features. In this paper, we attempt to tackle the video stabilization problem in a deep unsupervised learning manner, which borrows the divide-and-conquer idea from traditional stabilizers while leveraging the representation power of DNNs to handle the challenges in real-world scenarios. Technically, DUT is composed of a trajectory estimation stage and a trajectory smoothing stage. In the trajectory estimation stage, we first estimate the motion of keypoints, initialize and refine the motion of grids via a novel multi-homography estimation strategy and a motion refinement network, respectively, and get the grid-based trajectories via temporal association. In the trajectory smoothing stage, we devise a novel network to predict dynamic smoothing kernels for trajectory smoothing, which can well adapt to trajectories with different dynamic patterns. We exploit the spatial and temporal coherence of keypoints and grid vertices to formulate the training objectives, resulting in an unsupervised training scheme. Experiment results on public benchmarks show that DUT outperforms state-of-the-art methods both qualitatively and quantitatively. The source code is available at https://github.com/Annbless/DUTCode.


I. INTRODUCTION
Videos captured by amateurs using hand-held cameras are usually shaky, leading to unpleasing visual experiences. Moreover, unstable videos also increase the difficulty of downstream computer vision tasks [3], [4], [5], e.g., object tracking and visual simultaneous localization and mapping (SLAM). Video stabilization, which aims to remove the undesirable shake and generate stabilized videos, is a fundamental image processing task and has been actively studied.
Existing video stabilizers can be roughly classified into traditional approaches [6], [1], [7], [8], [9] and deep learningbased approaches [2], [10], as shown in Figure 1. Deep learning-based video stabilizers utilize deep neural networks (DNNs) to estimate the transformation from the unstable video to stable video [11]. To effectively learn stabilization using DNNs, they usually require a large number of paired unstable and stable videos for training. Unfortunately, in real world, paired unstable and stable data are difficult to collect due to spatial and temporal synchronization [2]. Moreover, considering that stability is a subjective visual experience that may depend on one's preference and focus, scene complexity, object moving speed, etc, there may be several reasonable stable videos that can explain a given unstable video, and vice versa. Consequently, collecting many stable videos for a given unstable video becomes more difficult. As an alternative, several stabilizers try to generate different unstable videos from one stable video using random projection matrices on each frame to mimic camera shake [12], [13]. However, the generation of pseudo-unstable videos always ignores the influence of depth variations and dynamic objects in the stable videos. In addition, the shake patterns from random projection are different from the real-world ones. Therefore, these methods introduce a domain gap between the generated unstable videos and the real-world unstable videos. There are some other methods perform two stage stabilization by prestabilization first using off-the-shelf stabilizers and then poststabilization using a new stabilizer [12], [13]. Although the pre-stabilization process could reduce the degree of shake that may facilitate the subsequent post-stabilization process, it also introduces distortion artifacts as well as large cropping, impeding the post-stabilization process.
On the other hand, traditional video stabilization methods [14], [15], [16], [17] stabilize videos without the requirement of paired data. Since it is difficult to directly measure the stability from the pixel values in a video, they resort to the camera trajectory as an alternative representation to account for stability and design stabilizers [1], [16], [17]. Specifically, these methods perform stabilization by first estimating the camera trajectory from the unstable video and then smoothing it to obtain a trajectory corresponding to an underlying stabilized video, which can be warped from the unstable one based on the transformation derived from the smoothed trajectory. These methods achieve impressive stabilization results. Nonetheless, since they rely on hand-crafted features to estimate the trajectory, the common textureless and occluded regions as well as dynamic objects in real world scenarios affect the representation ability of hand-crafted features, therefore degrading the stabilization performance [6], [1]. Since DNNs are well known for the powerful feature representation capacity, it seems promising to use DNNs to handle the aforementioned challenging issues when estimating the trajectory. Here comes the question, i.e., can we leverage DNNs for video stabilization without using any paired data?  To answer this question, we dive into the problem formulation of the above two categories of methods. Deep learning stabilizers use paired data since they have no prior inductive bias related to stabilization and the paired data are the only source for them to learn the implicit stabilization knowledge. As for the traditional stabilizers, they tackle this task by adopting a divide-and-conquer strategy, i.e., decomposing the stabilization task into two sub-tasks of estimating the trajectory and smoothing it 1 , where using the camera trajectory as a representation to account for stability indeed introduces a simple but strong prior knowledge, i.e., a smoothed trajectory implies to a stabilized video. In addition, both the estimation process and smoothing process can be performed based on the unstable video, where some other forms of prior knowledge related to stabilization is also exploited, e.g., the motion between pixels in correspondence should be small. Based on the above analysis, we attempt to tackle the video stabilization problem in a deep unsupervised learning manner in this paper. It borrows the divide-and-conquer idea from traditional stabilizers and leverage the representation power of DNNs to handle the challenges in real-world scenarios.
Technically, our deep unsupervised trajectory-based stabilization method named DUT consists of a trajectory estimation stage and a trajectory smoothing stage. In the trajectory estimation stage, instead of estimating a global trajectory, we alternatively estimate the grid-based trajectories like [14], [2]. Specifically, we first estimate the motion vectors of keypoints in each frame, which are more robust to the variations of noise, illumination, viewpoint, etc. The grid based motion are initialized by a novel multi-homography estimation strategy based on the keypoints and their motion. Then, we devise a novel motion refinement network to further improve the estimation accuracy. In this way, we obtain the grid-based trajectories via temporal association. In the trajectory smoothing stage, we devise a novel network to predict dynamic smoothing kernels to perform trajectory smoothing, which can well adapt to different trajectories due to the diversity of scene layout, depth, 1 The warping process to obtain the stabilized video is straightforward. and local motion. It is noteworthy that we exploit the spatial and temporal coherence of keypoints and grid vertices to formulate the training objectives, which supervise the networks to learn feature representations for each sub-tasks. In this way, our networks can be trained in an unsupervised manner.
The main contributions of this paper are threefold: • We propose the first deep unsupervised trajectory-based stabilization method (DUT) that leverages the representation power of DNNs to handle the challenges in real-world scenarios and is trained in an unsupervised manner without the requirement of paired training data.
• We propose a novel trajectory estimation method based on multi-homography estimation and motion refinement, which can address the issue of multiple planar motions and improve the accuracy and robustness of trajectory estimation regarding to noise, illumination change, occlusion, etc.
• We propose a novel trajectory smoothing method that can predict dynamic smoothing kernels for trajectory smoothing and well adapt to trajectories with different dynamic patterns.
• Our DUT outperforms state-of-the-art methods both qualitatively and quantitatively on public benchmarks while being light-weight and computationally efficient.

A. Traditional Methods
Traditional methods [20], [21] stabilize unstable videos by first estimating trajectories and then smoothing the trajectories for stabilization. CPW [6] utilizes structure from motion (SfM) [22] for 3D camera poses estimation, which is too fragile due to the difficulty in estimating exact camera poses and reconstruct scenes from unstable videos. [23], [24], [25] describe camera trajectories by tracking keypoints across all the frames and obtain the smoothed trajectory via optimization. [26] further introduces multiple-plane constraints into CPW and focuses on 3D based video stabilization. It validates the importance of recognizing different planes in video stabilization but still suffers from the fragile of 3D  The pipeline of our video stabilization framework, which contains three modules: keypoint detection module, motion propagation module, and trajectory smoothing module. The keypoint detection module utilizes the detector from RFNet [18] and optical flow from PWCNet [19] for motion estimation. (b) The motion propagation module aims to propagate the motions from sparse keypoints to dense grid vertices. It employs two separate branches of stacked 1d convolutions to embed keypoint location and motion features, generate attention vector and attended features to predict the motion vector for each grid vertex. (c) The trajectory smoothing module aims to smooth the estimated trajectories at all grid vertices by predicting smoothing kernel weights. Three 3d convolutions are stacked to process the trajectories both spatially and temporally. The estimated kernel and scale are multiplied to form the final kernel for iterative smoothing. motion estimation. Subspace [1] employs low-rank approximation for scene motion and camera motion decomposition and smooth the camera motion via polynomial path fitting. Geodesics optimization [8], bundled path estimation [16], epipolar geometry [27] are employed to increase robustness of the previous methods. However, these methods still need long-term tracked keypoints to estimate complete trajectories, which is challenging due to the appearance variance around keypoints. SteadyFlow [17] and MeshFlow [14] indicates that the frame-to-frame motions at specific locations can be used to describe the camera motion and free the stabilizers from updating tracked or missed keypoints across the whole videos. However, they still may fail in occluded and untextured areas due to the limited expression ability of hand-crafted features. Some specific hardware [15], [28], [29], [30], [31] is also employed to help the trajectory estimation, which imposes additional burden on the stabilization tasks. To this end, we propose to use DNN for accurate trajectory estimation and smoothing instead of relying on hand-crafted features for more robust video stabilization. With the strong expression ability of DNN models, the DUT stabilizer can better deal with unstable videos with large jitter.

B. DNN based Methods
Recently, DNN based video stabilizers have been proposed to regress unstable-to-stable transformation from data. Stab-Net [32] employs spatial transform network(STN [33]) to combines warping into the stabilization process for training using photometric loss. Similar strategy is also adopted in Xu et al. [11]. Although they can obtain stabilized videos, they need paired data for training and it is difficult for them to adapt to background depth variance and dynamic foreground objects, thereby producing distortion in these cases. To deal with the data shortage dilemma, some methods propose to generate unstable videos from stable videos by applying random generated projection matrices, e.g., Choi et al. proposes a neighboring frame interpolation method [34] based on generated unstable frames, which can generate stable videos but may introduce ghost artifacts around dynamic objects and buildings with large parallax. Such phenomena is caused by the ignorance of depth variance and dynamic objects during the generation of unstable videos. To mitigate the gaps between generated and real-life unstable videos, [12], [13] utilizes a two-stage stabilization pipeline by first utilizing traditional methods to pre stabilize the unstable videos and use DNN for further stabilization. Such design makes the performance of stabilizer is constrained both to the traditional methods' and the DNN models' performance especially in huge jitter cases. In contrast, our DUT method learns to stabilize unstable videos from the unstable videos themselves by progressively learning to estimate unstable trajectories and then smooth the trajectories. It directly and only uses the unstable videos for training, which alleviates the data shortage problem as the unstable videos themselves are easy to collect. Besides, due to there is no domain gap between the training data and real life unstable videos, DUT can generalize to a set of videos with good performance.

III. METHODOLOGY
As shown in Figure 2, given an unstable video as input, the DUT aims to generate a stable video by dividing this task into trajectory estimation and smoothing according to a divide-and-conquer strategy. It first estimates the grid-based trajectory by associating the motion of grid vertices, which are initialized with the motion of keypoints. The initialized motions are then further finetuned to more accurately describe the camera motion. Finally a deep learning based smoother stage is employed for a fast and stable trajectory smoothing by estimating dynamic smoothing kernels.

A. Motion Initialization
The motion initialization module (MI) aims to mitigate the effect of dynamic objects on the estimated camera motion, based on the motion of key points. Since the motion caused by camera shake is spatially coherent and the motion caused by dynamic objects is always different from this motion, we propose to enhance this coherence by using grid-based motion instead of pixel-based motion, which implicitly suppresses the effect of dynamic objects.
Denote the input video as where f i is the ith frame of the video and E is the number of frames, the motion initialization stage first extracts optical flow between adjacent frames using PWCNet [19], i.e., (1) PWCNet is a common choice in previous stabilizers [34] and we use PWCNet here for a fair performance comparison. The dense pixel-based motion contains noise from dynamic objects, occlusion, etc. Previous stabilizers [13], [17] alleviates such noise using heuristic rules, e.g., predefined thresholds to exclude the influence of dynamic objects' motion. Users need to carefully choose an appropriate threshold for each unstable video, which is not satisfactory for a stabilizer than can generalize to any unstable videos. Instead of using heuristic rules, we exploit the spatial coherence of motion caused by shake to alleviate the influence of dynamic objects by estimating grid-based motion from the pixel-based motion, using sparse keypoints as a middle presentation. We adopt the RFNet [18] as our keypoint detector to find the distinctive regions where the extracted optical flow is more reliable, i.e., where p i j is the jth detected keypoint on frame f i , L is the number of detected keypoints, which is set to 512 in this paper. Compared with the traditional keypoint detectors [35], [36], [37] and other deep learning based ones [38], [39], RFNet can efficiently and robustly produce high-resolution response maps for selecting robust keypoints since it adopts a multiscale and shallow network structure. For simplicity, we use the same symbol p i j to denote the keypoint location without causing ambiguity and the motion vectors of these keypoints can be expressed as m i j = OF i (p i j ).
After that, a simple yet efficient multi-homography estimation strategy is employed to initialize the grid-based motion using the keypoint-based motion. We uniformly divide each frame into a fixed set of grids, i.e., M × N grids in this paper, and use K-means to broadly segment each frame into two planes based on the keypoints motion, i.e., C c where c = 0 or 1 denotes the cluster index and Λ c i is the keypoint index set belonging to the cluster C c i . Note that when the number of keypoints in any cluster is less than a threshold, i.e., 20%×L = 102 in this paper, we merge it into the master cluster. Then each grid is assigned to one plane according to the majority cluster index of its neighboring keypints in Ω ik , which is defined as Ω ik = { j| d i jk 2 ≤ R}. Here, R is a predefined radius, i.e., 200 pixels in this paper. Plane-aware homography are estimated based on the segmented keypoints to initialize the grid based motion, i.e., H c i = Homo {p c i j |∀ j}, {m c i j |∀ j} and where v ik is the kth grid vertex on the ith frame and n ik is the motion of the grid vertex estimated by the initialization stage. It is worth noting that although we only use two homography estimation for motion initialization, our method can also handle frames containing more than two planes, as shown in Figure 4.

B. Motion Refinement
While the initialized motion can mitigate the effects of dynamic objects, the estimated motion is far from satisfactory as it does not estimate the camera motion accurately enough. To this end, we propose a CNN-based motion refinement stage to improve the accuracy of the estimated motion. Specifically, we first reconstruct the motion on detected keypoints based on the initialized gird vertex motion and calculates the residual between the reconstructed motion and the original motion estimated by PWCNet, i.e., The estimated motion should be accurate enough to make the residual ∆m i j small enough for each keypoints. However, as the keypoints still may cover dynamic objects, forcing the estimated motion to minimize the residual for each keypoint may introduce extra noise. Here we explore the motion coherence property again to distinguish the keypoints belong to static backgrounds or dynamic objects based on the assumption that the motion on static background should be consistent while on dynamic objects should not obey the global coherence. Such property is utilized to train the refinement model to find a better solution for accurate motion estimation.
Technically, we treat the sparse keypoints and dense grid vertices as two point clouds with motion and 2D location distributes and utilize several 1D convolutions to associate the keypoints' residual motion to each grid vertices to finetune the vertices' motion. Specifically, given ∆m i j and the distance vector d i jk between each grid vertex v ik and keypoint p i j , i.e., d i jk = p i j − v ik , k = 1, . . . , MN, the motion refinement stage predicts the residual motion vector ∆n ik for each vertex v ik , i.e., where MR (·) denotes the proposed motion refinement stage. The distance vectors and motion vectors of size [MN, L, 2] (after tiling) are fed into two separate encoders as shown in Figure 2(b The refined motion are then collected to formulate the trajectories on each grid and served as the input for the trajectory smoothing stage.

C. Trajectory Smoothing (TS)
Associating the motion vectors of corresponding grid vertices temporally formulates the trajectories that describe the camera motions, i.e., The key issue in the trajectory smoothing part is how to properly define the smoothness of the trajectory with respect to the different scene layouts, depths, etc. in each video. Here, we jointly use temporal and spatial coherence to characterize the properties of an ideal smooth trajectory based on grid vertices. First, we use temporal local smoothness to describe an aspect of the desired smoothness trajectory by constraining the motion between each temporal local region to be small, where Ω i defines the temporal local region of a temporal step i and T ik denotes the smoothed trajectory at v ik . w i j is the weight used to control the temporal local smoothness of the trajectory. However, if only the smoothness is considered, there will be a trivial solution because the smoothed trajectory may be a straight line, which is over-smoothed and cannot maintain the properties of the origin trajectory, e.g., direction, tendency. The constraints that the smoothed trajectory should not be far away from the origin trajectories are applied as where λ is introduced to balance the local smoothness term and the distance term. However, is it reasonable to require that every temporal local region be equally smooth? The local smoothness should vary dynamically according to the spatial properties of the trajectory itself and the grid, because the smoothed trajectory should not introduce additional distortions to the stabilized results. From this point of view, we further include a content preserving term in the objective function, since discarding spatial consistency can lead to distortion or content loss. The objective functions are detailed in Section III-D. Specifically, to predict the dynamic weights that can be used to smooth the trajectories, we adopt a novel 3D CNN structure based on the similarity of the trajectories. As shown in Figure 2 There are 12 weights estimated for each vertex since we use different weights for different dimensions and uses 3 as the temporal radius of a temporal region. The separately estimated dynamic balance term λ and dynamic smoothness weight term w d are further substituted in the temporal smoothness objective function L ts to dynamic determine whether a smoothed trajectory in a temporal region should be more smooth or more close to the origin trajectory to preserve the trajectories' direction. The smoothed trajectories are then iteratively solved in the iterative smoother by solving the following equation using the estimated dynamic weights as where λ ik and w ki j is the output of the CNN based smoother at each specific location k and temporal step i. In conclusion, the stabilization process can be summarized in Algorithm 1.

1) Motion Refinement:
For the estimation of target motion vector n ik of each vertex v ik , there are two constraints on it based on the temporal and spatial coherence. First, n ik should be close to the motion vectors of v ik 's neighboring keypoints. Second, the target position for each keypoint p i j can be calculated based on its motion vector m i j or its projection by the homogrpahy transformation of the grid that contains p i j , which can be derived from the four grid vertices' motion vectors [14], [40]. Based on the above two constraints, we formulate the objective function for the MR module as: where · 1 denotes the L1 norm, · 2 denotes the L2 norm, H i j is the homography of the grid containing p i j , λ m/v is the weight to balance the two terms. O i j is the occlusion mask to alleviate the influence of dynamic objects in the motion estimation process. Denote the reconstructed motion based on grid vertices as OF i , and the warped motion using OF i as ω( OF i ). The occlusion mask O i is calculated as, (8) where α 1 = 0.01 and α 2 = 0.5, as defined in [41], [42]. In addition to the keypoints-related constraints in Eq. (7), we also introduce a shape preserving loss L sp on each grid g im to enhance spatial coherence, i.e., where im denote four vertices of g im in the clockwise direction, their motion vectors are n 1 im ∼ n 4 im , R 90 (·) denotes rotation 90 • of a vector. L sp accounts for spatial smoothness and avoids large distortion in untextured regions. The final objective is: where λ s is loss weight that balances the two losses. Note that, although the above objective function can be solved using traditional solver, it takes much longer time than using our deep learning based solvers as it needs to iterate on over 1 million parameters on a standard 480p video. The idea of using DNN as a quick solver can be found in LSNet [43] and MeshSLAM [44].
2) Trajectory Smoothing: To help the CNN to predict appropriate dynamic parameters to facilitate the trajectory smoothing process, we add spatial coherence loss and content preserving loss in addition to the origin temporal smoothness loss L ts to characterize the desired smoothed trajectory, i.e., where L sp is the spatial shape preserving loss similar to the one used for MR in Eq. (9) where the target position of v ik after smoothing is: Moreover, since distortion in structural areas around keypoints affects the visual experience more than those in untextured areas, we introduce a content preserving loss L cp : where H i j is same as in Eq. (7). Assuming p i j is in the grid g im , then Bili (·) denotes a bilinear interpolation, i.e., Bili( im are the target positions of four vertices in g im after smoothing and calculated as Eq. (12). w 1 ∼ w 4 are the interpolation coefficients calculated based on the position of p i j relative to the four vertices of g im before smoothing, i.e., v 1 im ∼ v 4 im . L cp indeed preserves the shape of each grid containing keypoints, which probably has structural content. Note that the proposed CNN smoother can be trained using only unstable videos based on the objective in Eq. (11).
if There are multiple planes in frame f i then

IV. EXPERIMENTS
We evaluated our model on public benchmark dataset NUS [16] and compared it with state-of-the-art video stabilization methods, including deep learning based method [2], [34], [13] and traditional methods [1], [14]. Both quantitative and qualitative results as well as model complexity are provided for comprehensive comparison. Ablation studies of key modules in our model were also conducted.

A. Experiment Settings
Unstable videos from DeepStab [2] were used for training. Five categories of unstable videos from [16] were used as the test set. The metrics introduced in [16] were used for quantitative evaluation, including cropping ratio, distortion, and stability. Cropping ratio measures the ratio of remaining area and distortion measures the distortion level after stabilization. Stability measures how stable a video is by frequency domain analysis. All the metrics are in the range of [0, 1]. A larger value denotes a better performance.

B. Quantitative Results
We compared our method with representative state-of-theart video stabilization methods, including traditional methods: Subspace [1], Meshflow [14], and deep learning methods: DIFRINT [34], Yu et al. [13] and StabNet [2]. The quantitative Input Video SubSpace MeshFlow DIFRINT Our DUT Fig. 4: Visual comparison between our DUT and SubSpace [1], Meshflow [14], DIFRINT [34]. We present five kinds of videos for subjective evaluation, i.e., 1) moderate camera shake, 2) large dynamic objects, 3) small dynamic objects, 4) multiple planar motions, and 5) quick camera rotation and blur in the first row to the fifth row, respectively. It can be observed that our method can handle diverse scenarios compared with traditional stabilizers and deep learning based methods. More results can be found in the supplementary video.    , DUT produces less distortion and keeps more areas after cropping. Although StabNet also uses grid based warping while being trained with paired data from the deepStab, it does not achieve a balance between stability and distortion. By contrast, DUT simply uses unstable videos for training. As there are many possible stabilized videos that all look stable about the same, training with paired data that only provides one single stable instance for each sample may be biased. Besides, the supervisory signal from the stable video is less effective than those from the grid based pipeline, which have explicit relationships to stability, e.g., displacement derived from the grid motion and trajectory. Fourthly, compared with the interpolation based DIFRINT [34], DUT produces less distortion around the dynamic objects since it can handle them via robust keypoint based local grid warping. It is also interesting to find that although DIFRINT does not need cropping after stabilization, it does learn a zoom-in effect during stabilization and causes content missing problem in the stabilized results, as shown in Figure 4. Compared with Yu et al. [13], DUT does not need extra pre-stabilization step, and alleviates the limitation introduced by the pre-stabilization step, e.g., the traditional pre-stabilization makes the stabilizer does not perform well on unstable videos with quick rotation. Generally, DUT achieves the best performance regarding stability and distortion, which confirms the superiority of our trajectory based DNN stabilizer. Note that although we do not directly constrain the cropping ratio terms in the objective functions, DUT achieves impressive performance on the cropping ratio metric. We conjecture that it is because the estimated trajectory in DUT is more accurate with DNN and that the smoother part does not need to make large displacements to smooth the noise in estimated trajectories.

C. Subjective Results
Some visual results of different methods are shown in Figure 4. For videos containing dynamic objects (the 1st-3rd rows), DIFRINT [34] is prone to produce ghost artifacts around dynamic objects due to the neighboring frame interpolation, which is unaware of dynamic foregrounds. Subspace [1] may generate some distortion if the dynamic object is large as in the 2nd row. For scenes containing parallax and occlusion (the 4th row), Subspace also produces distortion due to the inaccurate feature trajectory. DIFRINT generates hollows since the interpolation cannot hallucinate the occluded objects. Meshflow [14] produces shear artifacts due to the inaccurate homography estimated from a single plane. DUT has no such drawbacks since it adopts an adaptive multihomography estimation method. For videos containing quick camera rotation and large blur (the 5th row), Subspace and Meshflow produce strong distortion due to the inaccurate trajectory estimation based on hand-crafted features. There is a dark patch in DIFRINT's result, which is caused by interpolating far away regions from adjacent frames to the inaccurate area of the current frame due to the quick camera motion. In conclusion, DUT is robust to different camera motion patterns and diverse scenes, owing to its powerful representation capacity of DNNs.

D. Ablation Studies
As we tackle the learning of video stabilization by a divide-and-conquer strategy, i.e., learning to estimate accurate enough trajectory and learning to smooth the trajectories with adaptative kernels, we will do ablation studies related to the two subtasks separately and discuss the results as follows.
1) Motion Refinement: Motion refinement is crucial for video stabilizers to provide accurate trajectory. To better investigate the influence of the Motion Refinement Module, we consider warping the neighboring frames to the current frame based on the estimated grid motions and calculate the error based on the warped frames and the current frame, since we do not have ground truth grid motion to directly assess the quality of motion estimation. Specifically, we consider two specific metrics for evaluation. First, if the grid motion is accurate, the warped frames should be close to the ground truth frame, i.e., there should be an identity homography between them. Therefore, we can calculate the Frobenius distance between the estimated homography and the identity homography as the first metric, i.e., "distance" in Table IV. Second, following [16], we calculate the ratio between the largest two eigenvalues of the homography as the distortion metric, i.e., "distortion" in Table IV.  The results are summarized in Table IV. MF denotes using the median filter for motion refinement where the initial motion and keypoints are generated by the proposed motion initialization module, while MR denotes using the proposed module for motion refinement. As can be seen, MR achieves better scores than MF in terms of both metrics, showing that the trajectory estimated by the proposed motion refinement module is more accurate than that using the median filter, which is attributed to the dynamic selection of reliable keypoints for motion refinement. In addition, based on the estimated trajectories in the above two settings, we use the same smoother (i.e., the proposed trajectory smoothing module) to obtain the stabilization results. The results are presented in Table V. It can be seen that using the trajectories estimated by the MR module leads to better stabilized results, which further validates the effectiveness of the proposed motion refinement module in obtaining accurate motion and trajectories.
To further demonstrate the effectiveness of our MR stage, we visualized the important supporting keypoints for some grid vertices selected by the MR stage in Figure 5. Note that we sorted the weights of all the keypoints generated by our MR stage for each grid vertex, and selected those with large weights as the supporting keypoints as shown in red circles in Figure 5. Other keypoints are marked as white circles. The grid vertex is marked as the blue point. For the grid vertex where there are few keypoints in its neighborhood, e.g., (a) and (c), the MR stage prefers choosing those keypoints which have similar motions to the grid vertex. For the grid vertex which is surrounded by rich keypoints, e.g., (b) and (d), the MR stage relies more on the keypoints which are nearby and have similar motions to the grid vertex. Compared with the median filter which treats each point in its neighborhood equally, our MR module can adaptively choose the supporting keypoints from all the keypoints for each grid vertex according to their distances and motion patterns. To investigate the impact of different terms in the objective function, we carried out an empirical study of the hyperparameters, i.e., the loss weights of the L1 loss term L 1 and the distortion constraint term L d . The objective function is reformulated as: We changed the weights of λ 1 and λ 2 and evaluated the MR module on test videos from the regular and parallax categories. Note that λ v and λ s were kept 1:1 according to their amplitudes on a subset of training data. The results are plotted in Figure 6. As can be seen, with an increase of λ 1 , both the distortion and distance metrics become marginally better for the regular category, while they become worse for the parallax category. Using a large λ 1 to emphasize L 1 in the objective function, the MR model acts like a median filter, which is not able to deal with multiple planar motions in the videos from the parallax category. By contrast, with an increase of λ d , our MR model achieves better performance for the parallax category in terms of both distortion and distance metrics. Generally, the MR module is robust to different hyper-parameter settings for the regular category. We set λ 1 as 10 and λ 2 as 40 in other experiments, i.e.,λ m = 10, λ v = λ s = 40. 2) Trajectory Smoothing: To provide a thorough analysis of the hyper parameters used in the trajectory smoothing stage, we separately investigate the influence of iterations and weights in the trajectory smoothing stage's objective functions regards the stability, distortion and cropping ratio. To evaluate the influence of different iterations of the TS module, we ran our smoother with different iterations, i.e., 5,10,15,20,30,40,50, 100 and plotted the average stability and distortion scores as well as the cropping ratios based on the NUS dataset in Figure 7.
It can be seen that the stability score increases rapidly at the beginning and then saturate after 15 iterations for all categories. For the distortion and cropping ratio, they decrease fast at the beginning and then decrease slowly. Besides, we plotted the smoothed trajectories using different smoothing iterations for a video from the Regular category in Figure 8, where the front view, profile view, and top view of the trajectories are plotted in (a)-(c), respectively. The unstable trajectory is shown in blue and smoothed trajectories are shown in different colors. As can be seen, with the increase of the iterations, the trajectory becomes more smoothing and deviates from the original trajectory rapidly at the beginning. Then, it changes slowly and converges after a few iterations. Generally, for different categories, there are slightly different critical points that can make a trade-off between these metrics. We choose 15 iterations for all categories according to the average performance to make a trade-off between different metrics as well as keep the computational efficiency of our method. The impact of different terms in the objective function of TS module is also investigated. We carried out an empirical study of the loss weights. The objective function was reformulated as: We changed the weights of λ 1 and λ 2 and evaluated the TS module on test videos from the regular and parallax categories. Note that λ s and λ c were kept 2:1 according to their amplitudes on a subset of training data. The results are plotted in Figure 9. As can be seen, with the increase of λ 1 , more stable results can be achieved for both categories while the distortion scores decrease, i.e., marginally for the Regular category but significantly for the Parallax category. It is reasonable since the distortion term contributes less to the objective function. When varying λ 2 , the stability score does not change a lot while the distortion scores first increase and then decrease at the late stage, especially for the Parallax category. The distortion term encourages the transformation in each grid to keep the grid shape. However, the grid around the boundaries between two planes may not be necessary to keep its shape due to the different planar motions. Thereby, a large distortion term will lead to distortions in these regions. Generally, the TS module is robust to different settings for  To further validate the effectiveness of the proposed trajectory smoothing module, we use the trajectories estimated by the MR module as input and apply both the proposed trajectory smoothing module and the traditional smoother [11] for stabilization. We iteratively smooth the trajectories using each smoother until obtaining similar stability scores. The total iterations are recorded, while the distortion scores and cropping ratios of stabilized videos are calculated. The results are presented in Table VI and Figure 10, where TR represents the traditional smoother while TS represents the proposed trajectory smoothing module. As can be seen, the proposed trajectory smoothing module uses fewer iterations to get similar stability scores, i.e., 15 v.s. 100, showing that the proposed TS module is more computationally efficient than the traditional counterpart. Besides, TS module gets higher distortion scores, demonstrating the superiority of the dynamic smoothing kernels in the proposed TS module.

3) Motion Refinement and Trajectory Smoothing:
To investigate the effectiveness of the proposed MR and TS modules, we use the same motion initialization module as ours in MeshFlow [14] to isolate the influence of motion estimation and compare it with our DUT. The results are presented in Table VII, where MeshFlow [14]+MI represents using the proposed motion initialization module for motion initialization, while keeping the median filter for motion refinement and the traditional smoother for trajectory smoothing. It can be seen that using the proposed MR and TS modules leads to better performance in terms of distortion and stability since the MR module can estimate more accurate trajectories while the TS module uses dynamic weights that can adapt to different trajectories. The running time, number of parameters, and number of computations of both traditional and deep learning methods are summarized in Table VIII. For a fair comparison, we run Subspace [1], StabNet [2], DIFRINT [34] and our DUT on a 640*480 video with 247 frames for 20 times and calculate the average per frame time cost. DUT has fewer parameters and computations, outperforming all other methods. Besides, it is much faster than Subspace and DIFRINT while being only slightly slower than StabNet. The optical flow calculated by traditional or deep learningbased methods may contain errors in some areas, e.g. dynamic objects or untextured regions. To evaluate the robustness of our DUT concerning inaccurate optical flow values, we randomly added noise on the optical flow map and ran our stabilizer based on it accordingly. Three types of noise were included in the experiments, i.e., Gaussian noise (denoting "G"), Salt and Pepper noise (denoting "SP"), and the value missing error (denoting "Blank"). Specifically, we added different types of noise on 10% regions randomly chosen from each optical flow map. We set four different levels of standard deviations for the Gaussian noise, i.e., 5% (G-5), 10% (G-10), 15% (G-15), and 20% (G-20) of the maximum flow values. We ran the experiment three times for each setting and calculated the average metrics. The results are summarized in Table IX. As can be seen, our model is robust to various types of noise and different noise levels, i.e., only a marginal performance drop is observed when there is noise in the optical flow map. It can be explained as follows. First, the detected keypoints are always distinct feature points that are easy to get accurate optical flow. Second, the keypoints are distributed sparsely on the image that some of them may not be affected by the randomly added noise. Third, our CNN-based motion propagation module can deal with the noise since it leverages an attention mechanism to adaptively select important keypoints by assigning high weights and aggregate their motions in a weighted sum manner. Thereby, it can be seen as an adaptive filter to deal with the noise (i.e., inaccurate optical flow values).  Evaluation of video stabilization is also a subjective task since different subjects may have different visual experience and tolerance on stability, distortion, and cropping ratio. Therefore, we carried out a user study to compare our model with the commercial software Adobe Premiere Pro CC 2019 for video stabilization. Note that, to the best of our knowledge, Adobe Premiere and Adobe After Effects adopt the same stabilizer with the default setting. First, we chose three representative videos from each of the five categories in the NUS [16] dataset to constitute the test videos. 25 subjects participated in our study with ages from 18 to 30. Each subject was aware of the concept of stability, distortion, and cropping ratio after a pre-training phase given some test videos and results by other methods. Then, the stabilized videos generated by our DUT and Adobe Premiere were displayed side-by-side but in random order. Each subject was required to indicate its preference according to the stabilized video quality in terms of the aforementioned three aspects. The final statistics of the user study are summarized in Figure 11. It can be seen that most users prefer our stabilization results than those by Adobe Premiere. We find that since Adobe Premiere uses traditional keypoints detectors, the detected keypoints in consecutive frames may drift, especially in the videos from the quick rotation category, leading to less satisfying stabilization results than our deep learning-based model. To further evaluate the generalization ability of the proposed DUT stabilizer, we collect different unstable videos from YouTube, which contain various camera motions, such as walking, running, driving, and climbing, and cover different light conditions. Different stabilizers and our DUT are evaluated on these videos. Both the quantitative and qualitative results are provided in Table X and Figure 12. It can be observed that the proposed DUT stabilizer still outperforms previous stabilizers on these videos, e.g., it achieves a gain of 0.07 stability score and 0.15 distortion score over the popular warping-based stabilizer StabNet [2]. These results confirm the generalization ability of DUT regarding different camera motions and light conditions, owing to its powerful representation ability 2 .

I. Limitations and Discussion
DUT decomposes the video stabilization task into learning to estimate accurate motion trajectories and smooth them for video stabilization in an unsupervised manner. Although the pipeline is similar to traditional stabilizers, leveraging deep neural networks yields robust and fast stabilization results. However, there are some hyper-parameters that can be further carefully tuned, e.g., the number of smoothing iterations for different types of videos and the multi-planar thresholds in the motion refinement module. For example, although the DUT with default parameters generally performs well, it may cause distortion on some videos, as shown in Figure 13. To address this issue, more research efforts on devising efficient strategy for adaptive hyper-parameter setting or leveraging the attention mechanism for a more robust solution, can be made in the future work. Besides, since the satisfactory video stabilization result of a shaky video is not unique, how to objectively evaluate the performance of video stabilizers by either full-reference quality assessment [45] or non-reference quality assessment [46] remains an open challenge.
It is noteworthy that in this paper we focus on improving the performance of warping-based video stabilization methods, especially regarding stability and distortion of their stabilized Fig. 12: Some stabilization results of the DUT on videos from different scenes regarding lighting conditions and camera motions. The odd-numbered rows are the input frames, while the even-numbered rows are the stabilized results of DUT. The first and second video clips are taken on cloudy days and at nightfall, respectively, while the third video clip is taken during climbing, and the fourth video clip is taken during driving. More results are shown in the supplementary video. videos. To this end, we propose the deep neural networkbased DUT model as well as explore an unsupervised training scheme. Nevertheless, it also suffers from the low crop ratio issue like other warping-based methods due to content loss. The reason is that the current frame may not have the candidate pixels for unstable to stable warping, i.e., resulting in a blank area that should be cropped in the final stabilized video for a better visual experience. Inspired by the success of DIFRINT, we believe that such pixels can be traced from neighboring frames for unstable to stable warping. We leave it as the future work to seek a full-frame video stabilization solution.

V. CONCLUSION
We propose a novel method DUT for video stabilization by decomposing video stabilization into estimating and smoothing trajectories with DNNs for the first time. DUT learns to do video stabilization step by step in an unsupervised manner. Both quantitative and qualitative results on the popular NUS dataset confirms the performance of the proposed DUT methods. Moreover, it is light-weight and computationally efficient. More effort can be made to improve the performance of DUT further, e.g., devising an adaptive multi-plane segmentation network. We hope such a work can provide a new perceptive on unsupervised video stabilization learning, regards the difficulty in collecting paired stable-unstable videos.