{'fraction_non_alphanumeric': 0.030168946098149636, 'fraction_numerical': 0.008145615446500403, 'mean_word_length': 5.003320253546635, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multiple Combined Constraints for Image Stitching

Several approaches to image stitching use different constraints to estimate the motion model between image pairs. These constraints can be roughly divided into two categories: geometric constraints and photometric constraints. In this paper, geometric and photometric constraints are combined to improve the alignment quality, which is based on the observation that these two kinds of constraints are complementary. On the one hand, geometric constraints (e.g., point and line correspondences) are usually spatially biased and are insufficient in some extreme scenes, while photometric constraints are always evenly and densely distributed. On the other hand, photometric constraints are sensitive to displacements and are not suitable for images with large parallaxes, while geometric constraints are usually imposed by feature matching and are more robust to handle parallaxes. The proposed method therefore combines them together in an efficient mesh-based image warping framework. It achieves better alignment quality than methods only with geometric constraints, and can handle larger parallax than photometric-constraint-based method. Experimental results on various images illustrate that the proposed method outperforms representative state-of-the-art image stitching methods reported in the literature.


INTRODUCTION
Traditional image stitching methods estimate a global homography transformation to align image pairs [1]. The underlying two conditions (concentric condition and planar condition) are difficult to meet in practical applications. Recently spatially-varying motion model and mesh-based warping model are proposed to improve image stitching quality, especially for image pairs with large parallax. Compared with the global homography model, the higher degree of freedom makes them more flexible to handle large parallax, but also makes the model estimations more difficult. A lot of constraints are applied to restrain the process of model estimation, which can be roughly classified into two categories: geometric constraints and photometric constraints. Geometric constraints further include the point constraints and the line segment constraints.
A lot of methods use point features in the overlapping region as constraints to stitch image pairs [2,3,4,5,6,7]. Gao et al. [2] divided the image content into a ground plane and a distant plane. It estimated a two-homography model by point matches and align image pairs by fusing these two homographies according to a weight map. Lin et al. [3] employed two sets of unmatched point features to jointly estimate the point correspondences and the spatially-varying affine transformation model simultaneously. Zaragoza et al. [4] extended the spatially-varying affine transformation to spatiallyvarying homography transformation. It handled stitching parallax by allowing local non-projective deviations apart from global homography transformation.
Point features are usually distributed with spatial bias [8] and are not sufficient enough to stably estimate the local warping model in some extreme cases (e.g., low-texture images). Therefore, a lot of methods [8,9,10,11,12,13,14] proposed to impose more constraints by line features. Joo et al. [8] proposed the line guided moving DLT (L-mDLT) method, which estimated a spatiallyvarying homography model with line correspondences. Similarly, Li et al. [11] proposed a mesh-based model by considering both point and line correspondences. Different from the above two methods who referred to the line constraints as data-term, In [12,9], line features are used as smoothness-term to preserve the line straightness before and after image warping.
Using the above two geometric constraints for image stitching is straightforward, but it strongly relies on abundant and robust results of feature extraction and matching, which are difficult to be ensured for images with complex structures. The photometric constraint is a good alternative that has been widely used in the problem of optical flow estimation. Recently, Lin et al. [15] proposed to employ the photometric constraints to image stitching. It combined the advantage of dense photometric alignment with the efficiency of mesh-based image warping, and it obtained better alignment quality than geometric-constraint-based methods in both textured and low-textured images. However, the optimization process with photometric item was easily trapped into local minimum and the proposed method was not suitable for images with large parallax.
In this paper, we propose to augment the basic contentpreserving warping (CPW) [16] framework by combining the photometric constraints with the geometric constraints.
We observe that geometric and photometric constraints are actually complementary. On the one hand, geometric constraints are usually spatially biased and are insufficient in some extreme scenes, while photometric constraints are always evenly and densely distributed. On the other hand, photometric constraints are not suitable for images with large parallax, while geometric constraints imposed by feature matching are more robust to handle image parallax. We combine them in an unified CPW framework to achieve better alignment quality than geometric-constraint-based methods and handle larger parallax than photometric-constraint-based method. Abundant experiments demonstrate that our proposed method outperforms representative state-of-the-art image stitching methods.

OVERVIEW OF THE PROPOSED SCHEME
The proposed image stitching method adopts the two-stage pipeline which has been widely used in [10,11,15]. Firstly, a global parametric model is estimated from feature correspondences to roughly stitch image pairs. Next, the content-preserving warping (CPW) [16] is applied, which is a mesh-based model and can further align images in the overlapping region. Let I1 and I2 are a pair of images to be stitched, major steps of the proposed method include: (1) Estimate the global homography using point and line correspondences and apply global image warping. Specifically: • The point and line correspondences are firstly extracted from image pairs. In order to obtain point correspondences, we simply use SIFT [17] feature implemented by VLFeat [18]. As for line segment, we adopt LSD [19] to detect line segments and employ Line-point invariant (LPI) [20] matching to obtain line correspondences. • Secondly, we follow the process described in [11] to parameterize point and line features. • Thirdly, the global homography H is estimated using direct linear transformation (DLT) with random sample consensus (RANSAC) [21]. • Lastly, I1 and I2 are globally stitched together.
The global alignment step would be beneficial to final stitching result in two aspects: For one thing, the global homography estimated from point and line features achieves better global alignment quality than the one estimated only from point features. For another, the process of RANSAC picks reliable matchings out from initial feature correspondences, which will be utilized in the next contentpreserving warping.
(2) Apply CPW with multiple combined constraints over global aligned image pairs to locally align images. Let Is and It be the image pair after global alignment. The CPW achieves the local alignment by warping Is to It. Similar to most existing methods, we construct a cost function considering the alignment quality as well as the smoothness of the mesh-based model and optimize it to obtain optimized positions of mesh vertexes.
• Cost function. We propose to introduce geometric constraints and photometric constraints into CPW framework simultaneously. Overall, three data terms and two smoothness terms are considered in the cost function, which are point correspondence term (Section 3.1), line correspondence term (Section 3.2), photometric term (Section 3.3), similarity transformation term (Section 3.4) and line collinearity term (Section 3.5). • Optimization. As the photometric constraints are related to image content, it will get updated after each CPW. Therefore, we perform the process of CPW multiple times until the mesh becomes stable. In order to stitch images with large parallax, similar to [15], we also adopt a coarse-to-fine optimization scheme on Gaussian pyramid images.
After the optimization, local homography can be inferred from the transformation of each mesh vertex, based on which Is are warped toÎs to be locally aligned to It.
(3) Blend the target image It and the warped source imageÎs linearly to obtain the resultant stitching image. In this paper, we simply use linear blending to illustrate the misalignments and make comparisons with other algorithms. Actually, many advanced blending method (e.g., multi-band blending [22]) can be employed to get better stitching results. We will describe the proposed CPW with multiple combined constraints in more detail in Section 3.

MULTIPLE CONSTRAINTS FOR CONTENT PRESERVING WARPING
Images after global alignment have been roughly stitched together. In the following content-preserving warping (CPW) step, we first divide Is into a m × n regular mesh (m = n = 32 in our implementation), where the original coordinates of mesh vertexes are denoted as V. The objective of CPW is to obtain the optimized positions of mesh vertexesV. In order to accomplish this target and estimate the mesh-based model robustly, we propose to combine the geometric constraints with the photometric constraints in the cost function of CPW. Specifically, point correspondence, line correspondences and photometric terms are three data terms to ensure that the optimized mesh can eliminate the misalignment in the overlapping region. Similarity transformation and line collinearity terms are two smoothness terms that are used for shape preservation.

Point Correspondence Term
We restrict the matching feature points between Is and It to be warped to close positions to improve local alignment quality in image overlapping region. Let p and p ′ be a pair of matching points on Is and It respectively. We first denote p with four vertexes of the quad containing p by bilinear interpolation. Let be the corresponding four vertexes, and wp = [ω 1 p , ω 2 p , ω 3 p , ω 4 p ] T be their bilinear interpolation coefficients. p thus can be expressed as p = Vpwp. We assume that the interpolation coefficients wp is consistent before and after CPW. We therefore construct the point correspondence term by summing alignment errors over all matching feature points.
where pi and p ′ i denote the i-th pair of matching points.

Line Correspondence Term
Similar to the point correspondence term, we restrict the matching lines to be close after local warping. Suppose that l = [a, b, c] and are a pair of matching lines of Is and It. In order to measure the alignment error between l and l ′ , we uniformly sample key points along l and require the distance from all key points to l ′ to be minimized. Therefore, the line correspondence term is defined as: where p j,k denotes the k-th key point of j-th pair of matching lines. Vp j,k are four mesh vertexes of the quad enclosing p j,k and wp j,k are the corresponding bilinear interpolation coefficients.

Photometric Term
Above two geometric constraints are widely used to mesh-based model estimation. Recently, Lin et al. [15] adopt a photometric constraint to align images. Their proposed photometric constraint is defined as: where pi is a set of uniformly sampled points in overlapping region. Is(·) and It(·) denote image intensity of source image and target image respectively. They parameterize the offset τ (pi) of sampled point pi using offsets of four mesh vertexes enclosing pi and relate the minimization of Eq. 3 to mesh-based optimization. We append a gradient-based component to Eq. 3 to beyond the illumination constancy to form our photometric constraint term: where Epg is the gradient component of our photometric constraint and λ is the weight to balance intensity and gradient components (λ = 1 in our implementation). Epg is defined as: It is to be noted that, in order to achieve rotational invariance [23], we set Gs,t = ∇Is,t rather than Gs,t = ∇Is,t.

Similarity Transformation Term
We define our first smoothness term similar to the one of [16], which constrains the similarity transformation for each quad. For each quad [v1, v2, v3, v4] in current mesh grid, it can be divided into two triangulations △v1v3v2 and △v4v3v2. Vertex v1 of △v1v3v2 can be represented as: where u and v can be computed from original positions of v1, v2 and v3. To constrain each quad to undergo a similarity transformation, for the warped triangulation △v1v3v2,v1 should be ensured to be represented byv2 andv3 using the same local coordinates (u, v) that computed from original positions. Therefore, the overall similarity transformation term is defined as: where Nt is the total number of triangulations in the mesh grid.

Line Collinearity Term
We construct another smoothness term based the extracted line segments. Unlike the cost function proposed in [11], in which only matched lines are utilized to constrain the cost function, we use the matched lines as one of our data term (Section 3.2) and use all extracted lines (matched and unmatched) to define our line collinearity term. Similar to Section 3.2, for each line segment l on Is, its two endpoints are denoted as pe and ps. We uniformly sample key points along l and compute its 1D coordinate u in the local coordinate system defined by pe and ps. To preserve the straightness of line segment, each key point is supposed to be represented by the same local coordinate after mesh-based warping. Therefore, the line collinearity constraint term is defined as: where p i e and p i s are two endpoints of i-th line segment. p i j is the j-th key point of the i-th line segment. N i p denotes the number of key points that the i-th line contains and N l is the total number of extracted line segments of Is. The key point and endpoints p i e and p i s are further parameterized by the mesh vertexes using bilinear interpolation, which is similar to the process described in previous section.

Overall Cost Function
We append a weight factor to each constraint term to form our final cost function: E = αE f + βE l + γEp + δEc + ηEs. (9) In Eq. 9, η is set to 0.2 and another four weights are all set to 1.0 in our implementation. The overall cost function is quadratic and can be easily minimized by any sparse linear system. We minimize the cost function defined by Eq. 9 multiple times and after each minimization, we update the photometric term Ep and continue to the next minimization. We consider the optimization to have converged if the average change of vertex positions between adjacent two optimizations is smaller than one pixel. Finally, we use the optimized mesh to warp Is toÎs locally and blend it with It to get the stitching result.

EXPERIMENTAL RESULTS
In order to demonstrate the superiority of our proposed image stitching method, we quantitatively compare it with three stateof-the-are methods: point-correspondence-based APAP [4], dualfeature-based DF-W [11] and photometric-based MPA [15]. After that, we conduct another experiment to qualitatively compare the stitching results produced by our method with different constraint combinations. for three compared methods on image pairs. APAP: as-projective-as-possible method [4]; DF-W: dualfeature method [11].

Quantitative Comparison
To quantitatively measure the alignment accuracy of image stitching, we adopt the same measurement in [11,15] to compute the stitching quality of each pair of images. Specifically, we compute the RMSE of one minus normalized cross correlation (NCC) over a local w × w window for all pixel in the overlapping region to obtain the stitching quality metric: where Ω denotes the overlapping region ofÎs and It. N is the number of pixels in Ω. ps and pt are the corresponding pixels in Is and It respectively. Evaluation on Low-Texture Images. To demonstrate the performance of our proposed method on stitching images with low textures, as shown in Fig. 1, we evaluate our method on the dataset provided by [11]. We compare the stitching quality of results produced by APAP, DF-W and our proposed method. For APAP and DF-W, the results are directly obtained from [11]. Table. 1 presents the comparative results. As we can see, DF-W adopts matching points and lines to constrain the stitching process, which has better results compared with APAP in most cases. Our proposed method combines the photometric constraints with above two geometric constraints and outperforms APAP and DF-W. As for image pair bench and road, APAP performs better than DF-W because limited line features can be extracted from these two image pairs with wide baseline, while our method still has the best performance because the combination of geometric and photometric constraints. Evaluation on Large-Parallax Images. We evaluate the performance of our proposed method on stitching large-parallax images  [15]. Initial: Our global alignment method.
on the dataset presented in Fig. 2. It consists of 8 pairs of images with large parallaxes, which are collected from publicly available datasets [4,7,12]. We compare the alignment accuracy of results produced by APAP, MPA [15] and our proposed method. Table. 2 shows the comparative results. MPA uses dense photometric constraints to restrain a mesh-based stitching model and gets better alignment accuracy than APAP. However, similar to the problem of optical flow estimation, optimization with photometric constraint is easily trapped into local optimums and may not work well on images with large parallax. Our proposed method combines photometric constraints with geometric constraints to handle large parallax and usually has lower stitching errors than MPA.

Qualitative Comparison
We design a comparative experiment to demonstrate the superiority of our proposed method by comparing stitching results produced by methods with three different constraint combinations: (1) geometric constraints; (2) geometric constraints followed by photometric constraints; (3) geometric constraints combined with photometric constraints. It is to be noted that the second combination is different from the third one (our proposed). The second combination imposes photometric constraints after geometric constraints and refers to the photometric item as a post-processing tool, which is similar to [15]. Fig. 3 presentsthe qualitative results. Mesh-based warping with point and line constraints fail to get satisfactory results on two selected image pairs where the extraction and matching of line segments are difficult for complex image structures. The following photometric-based post-process can improve the alignment quality locally but the improvements are very limited. As the photometricbased optimization is sensitive to initial values and is easily trapped into local minimum, in our experiment, this post-process sometimes may deteriorate the stitching result. In contrast, the results obtained by our proposed method always have good alignment quality in the overlapping region, because photometric constraints make a complement to geometric constraints in the joint optimization process.

CONCLUSION
In this paper, we propose a mesh-based warping method for image stitching, which combines the geometric constraints and the photometric constraints. The photometric constraints can always extracted from image content evenly and densely, and the geometric constraints are more robust to handle large parallax. Jointly utilize these two kinds of constraints in an unified content-preserving warping framework makes them complementary. A lot of quantitative and qualitative experiments demonstrate the superiority of our proposed image stitching method.