{'fraction_non_alphanumeric': 0.0333889816360601, 'fraction_numerical': 0.009398380016076176, 'mean_word_length': 4.737318197942533, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semi-Global Stereo Matching with Surface Orientation Priors

Semi-Global Matching (SGM) is a widely-used efficient stereo matching technique. It works well for textured scenes, but fails on untextured slanted surfaces due to its fronto-parallel smoothness assumption. To remedy this problem, we propose a simple extension, termed SGM-P, to utilize precomputed surface orientation priors. Such priors favor different surface slants in different 2D image regions or 3D scene regions and can be derived in various ways. In this paper we evaluate plane orientation priors derived from stereo matching at a coarser resolution and show that such priors can yield significant performance gains for difficult weakly-textured scenes. We also explore surface normal priors derived from Manhattan-world assumptions, and we analyze the potential performance gains using oracle priors derived from ground-truth data. SGM-P only adds a minor computational overhead to SGM and is an attractive alternative to more complex methods employing higher-order smoothness terms.


Introduction
Semi-Global Matching (SGM) is a widely-used stereo matching technique introduced by Hirschmüller [24,26]. It combines the efficiency of local methods with the accuracy of global methods by approximating a 2D MRF optimization problem with several 1D scanline optimizations, which can be solved efficiently via dynamic programming. It has been shown that SGM is a special case of message-passing algorithms such as belief propagation and TRW-T [12].
SGM has had significant impact, and the method is widely used in real-world applications, including 3D mapping, robot and UAV navigation, and autonomous driving [27,35,4]. SGM is also present in popular computer vision libraries such as OpenCV and has been implemented in hardware via FPGAs [19] and on GPUs [3].
While the method works well for aerial imagery and textured outdoor scenes, it works less well for indoor scenes with large untextured regions. The reason is that the algorithm employs a simple first-order smoothness assumption that favors fronto-parallel surfaces. It therefore tends to hallucinate fronto-parallel patches on weakly-textured slanted surfaces, which fail common consistency checks and result in large "holes" in the reconstructed surface, in particular when matching high-resolution images (Fig. 1d).
In this paper we propose a simple extension to the SGM algorithm, SGM-P, that utilizes precomputed surface orientation priors. Such priors favor different surface slants in different regions of the disparity space, and are implemented via local adjustments to SGM's transition penalties. The basic idea is to render the prior surface in the 3D disparity space and store the locations of discrete dis-parity steps in an offset image (Fig. 1e). SGM's smoothness term is then modified so that the zero-cost surface follows these steps and thus stays parallel to the prior surface. Arbitrary surfaces, not just planes, can be used as orientation priors, and our algorithm also supports multiple surface priors at different depths via an offset volume. Our method acts as a soft constraint during matching and only adds a minor computational overhead. In this paper we demonstrate that even simple surface priors can yield significant improvements in difficult indoor scenes containing slanted surfaces with weak texture (see Fig. 1f). Importantly, our experiments show that in the absence of such difficulties the performance never significantly decreases.
The SGM-P algorithm is agnostic as to the source of the priors, which could be computed in many different ways. For instance, surface priors can be derived from matched sparse feature points via triangulation [20] or plane fitting [42]. Planes (or other parametric surfaces) can also be fitted to disparities estimated at a lower resolution, which is one method we explore in this paper. Alternately, surface orientation priors could stem from domain knowledge (e.g., expected ground plane orientation in autonomous driving [28]), from semantic analysis [15,16,2], or from vanishing point analysis and Manhattan-world assumptions [41,43,17,32,36], which we also explore in this paper. Finally, surface priors could be derived from other sensors with lower resolution (e.g., commodity depth cameras) to aid high-resolution stereo matching in untextured regions.

Related work
Stereo matching is one of the oldest and most-thoroughly studied problems in computer vision [40,10]. Methods can generally be categorized into local and global methods [40]. Both types of methods make smoothness assumptions about the observed world; the former implicitely (e.g., by aggregating a matching cost over a local window), and the latter explicitly via a smoothness term that imposes a prior on the surfaces in the world. The simplest and most common smoothness assumption is first order and states that two neighboring pixels most likely have the same depth. This is assumed in both simple window-based methods such as SSD and pixel-based global MRF approaches [9]. A firstorder smoothness assumption introduces a fronto-parallel bias. This is not a problem when there is sufficient texture in the scene, but causes errors on untextured slanted surfaces, which is often problematic for indoor scenes (see Fig. 1d).
Many approaches have been proposed that can handle slanted surfaces. Woodford et al. [47] show how secondorder smoothness terms can be efficiently optimized via QBPO. Li and Zucker [33] derive smoothness models for slanted and curved surfaces using differential geometry. Bleyer et al. propose surface stereo [7] and object stereo [8] algorithms in which the scene is modeled with planes or splines, and PatchMatch stereo [6], in which local estimates of disparities and surface slant are propagated to neighboring regions. Sinha et al. [42] run local plane sweeps around disparity planes estimated from sparse feature matches.
Plane-sweep stereo with a preferred plane orientation was proposed by Collins [11], and subsequently extended to multiple plane orientations, Manhattan-world scenes, and piece-wise planar scenes [18,17,43,30].
A more recent trend is to formulate stereo matching using continuous MRF frameworks [48]. While PatchMatch stereo [6] was a greedy algorithm, follow-up work such as PMBP [5] incorporates regularization. Other work employs MRFs with continuous labels, using fusion moves for optimization [34,44]. Several recent stereo algorithms [21,22,50] use surface normal priors derived from single images [15,31,16,2] These methods utilize continuous optimization and require minimization techniques such as primal-dual methods or linear programming. These ideas cannot be directly incorporated into SGM or another discrete MAP inference framework. Our proposed algorithm offers a simple and efficient alternative to such complex approaches, and contributes a practical way of imposing surface orientation priors amenable to discrete optimization.
In the context of SGM, several improvements have been proposed. The CSGM method by Hirschmüller [25] estimates the disparities in untextured regions by fitting planes to adjacent textured pixels. In contrast to such postprocessing, our method incorporates orientation priors during the matching. Hermann et al. [23] suggest an approximate second-order smoothness term for SGM but do not demonstrate a clear performance gain. A hierarchical approach to SGM [37,46] aims to reduce ambiguities and runtime by restricting the search range based on SGM results computed at a lower resolution. We use a similar idea as one possible mechanism to derive surface orientation priors. While hard constraints such as search-range reduction can cause fine detail to be missed, in our case we only use the result from a coarser resolution to obtain a soft constraint on surface orientations.
Finally, our priors could also be added to other recent modifications of SGM, for instance MGM [14], which integrates results from multiple directions. Similarly, our technique is orthogonal to recent advances in matching cost learning by CNNs [49] as we show in experiments below.

Algorithm
We first review the SGM algorithm, then describe our proposed extension SGM-P.

SGM
The Semi-Global Matching (SGM) algorithm [26] is an efficient technique for approximate energy minimization of where C p (d) is a unary data term that represents the cost of matching pixel p at disparity d ∈ D ={d min , . . . , d max }, and V (d, d ) is a pairwise smoothness term that penalizes disparity differences between neighboring pixels. Specifically, V implements a first-order smoothness assumption, Instead of minimizing the 2D MRF, which is NP-hard, SGM efficiently minimizes a 1D version of Eqn. 1 along 8 cardinal directions r via dynamic programming [26]. For each direction r, SGM computes an aggregated matching cost L r (p, d) recursively defined from the image boundary: The 8 aggregated costs are summed at each pixel, yielding an aggregated cost volume whose per-pixel minima are chosen as the winning disparities d p = arg min d S(p, d).
Drory et al. [12] observe that the sum of the 8 individual minima of L r (p, d) is a lower bound on the minimum of the aggregated cost S(p, d) at each pixel p, and define an uncertainty measure U p as the difference between the two: The intuition is that U p will be zero at locations where the 8 minimum-cost paths agree, e.g., in textured regions where incorrect disparities have high unary costs C p . In untextured regions, however, multiple disparities will have similar unary costs, and the 8 individual minima of L r will likely occur at different disparities, in particular on slanted surfaces. We use U p in our experiments to plot disparity error as a function of uncertainty, and also in Fig. 1 to select high-confidence matches.

SGM-P
In order to utilize surface priors, the basic idea is to modify SGM's smoothness penalties to favor surfaces with the expected surface slant. The problem is that we cannot represent fractional surface slants in algorithms that use discrete disparities, such as SGM. Thus, the key idea is to rasterize the disparity surface, i.e., render it in the 3D pixel grid, and record the locations of the steps in the discretized disparity values. At these locations we then shift SGM's smoothness penalties V so that the zero-cost transitions coincide with these steps. See Fig. 2 for illustration. We first discuss the case where we have only one orientation prior per pixel.

2D orientation priors
Assume we are given a real-valued disparity surface prior S whose orientation at any given pixel p we would like to encourage across all possible disparities (Fig. 2a).
Let r be the current "sweeping direction" of SGM. Given a pixel p and its successor p = p + r, we rasterize the surface S to integer disparitieŝ and compute the discrete disparity steps (or jumps) We replace the original smoothness penalty V with a new function V S that incorporates the disparity jumps: At pixels where the value of j is nonzero, V S favors taking that disparity step and encourages the disparity surface to stay parallel to S. We can efficiently compute V S by storing the jumps j in an offset image (Fig. 2b). Since the jumps depend on the direction r, four different offset images are needed, one for each pair of opposing directions. However, we do not need to precompute and store all four offset images simultaneously. Instead, we only need to storeŜ. Before running scanline optimization on each pair of opposing directions, we generate the associated offset image using Eqns. 7 and 8. When reversing the direction, we flip the signs of the offsets.

3D orientation priors
We now relax the requirement of a single orientation prior per pixel and allow multiple overlapping surface hypotheses at different depths. As before, each surface should only act as an orientation prior, but should influence all nearby points (Fig. 2c). Assume that at pixel p we have K disparity surfaces {S p k }, k = 1 . . . K. For a given disparity d, we find the closest surface in terms of disparitỹ Then, we rasterize it to integer disparitieŝ and again compute the discrete disparity jumps between adjacent pixels. This time, however, they depend on d: The new smoothness penalty term now becomes Fig. 2d illustrates this disparity-dependent orientation prior. In order to compute V S efficiently, we store the precomputed values ofS p d for all p and d in an auxiliary volume. This can be seen as a 1D discrete Voronoi diagram for each pixel along the disparity axis in the volume. The values in each column can be computed efficiently with a forward scan followed by a backward scan after rendering all disparity surfaces into the column at this pixel.

Surface normal priors
So far we have considered using disparity surfaces as orientation priors. We now focus on the case where we are given a surface normal map as prior, obtained for instance via photometric stereo [13] or from Manhattan-world priors [32]. Such normal maps can be used in our SGM-P algorithm, though the situation is more complex than one might expect. First, we cannot use orientations directly, since SGM-P requires an actual surface that can be rasterized. Thus, the normal map must be integrated. Second, we need to distinguish between scene space (in Euclidean world coordinates) and disparity space. While a surface normal map can be considered a 2D prior since it encodes a single surface orientation per pixel, this orientation is given in scene space. As we show below, when converting to disparity space under a perspective projection model, the surface orientation becomes depth-dependent and results in a 3D disparity orientation prior, requiring an offset volume representation. See Figure 3 for illustration.
We now derive the relationship between surface normals in scene coordinates and the orientation of disparity surfaces. Given the surface normal vector (n p x , n p y , n p z ) T at pixel p, the equation of the tangent plane of the surface at p in scene space coordinates (x, y, z) is n p x x + n p y y + n p z z = h p . Here, h p encodes the plane's unknown depth. Under perspective projection we have x = uz/f and y = vz/f for image coordinates (u, v) and a camera at the origin with focal length f . Substituting these values we obtain For stereo pairs we have z = bf /d, where b and d are the baseline and disparity respectively. Substituting z into Eqn. 14 we obtain the disparity plane equation Note that the disparity plane orientation depends on h p , which encodes the depth of the tangent plane in scene space. Therefore, a scene plane with fixed orientation but unknown depth yields a family of disparity planes whose orientation depends on the associated disparity.
In order to use surface normal priors in SGM-P, they first need to be integrated into a surface. We do this integration in scene space under a perspective projection model using a least-squares approach [45] (see the supplementary materials for more details). The result is a z-surface in scene space coordinates, initially at an arbitrary depth. We scale this surface by an appropriate sequence of scale factors (Fig. 3b) and convert to d to arrive at roughly equallyspaced d-surfaces covering the full disparity range (Fig. 3c). Finally, we construct an offset volume from this family of disparity surfaces as described in the previous section, resulting in a 3D orientation prior with varying disparity surface slants. (Fig. 3d).

Experiments
We now demonstrate the utility of our new algorithm by comparing a baseline SGM implementation with various version of SGM-P employing different types of priors. For a fair comparison we use the same matching cost and smoothness weights across all versions of the algorithms.
To compensate for global and local rectification errors, we first robustly fit a global model y = ay + b to matched feature points and warp the right image accordingly before computing the matching costs. During matching, for each horizontal disparity, we evaluate matching costs corresponding to vertical disparities of {-1, 0, +1} pixels and select the smallest of the three costs.
For SGM's unary data term we use negated and truncated normalized cross correlation (NCC): where NCC(p, d) compares 5×5 grayscale image patches centered at p and p − (d, 0) T in the left and right image, respectively. Image intensities are in the range [0, 255]. We add a small value = 1.0 to the NCC denominator to suppress the effect of noise in untextured regions. We scale C p (d) by 255 and round it to the nearest integer. We can use unsigned shorts for SGM's aggregated costs, which reduces the memory overhead.
We use NCC as matching cost since it is commonly employed in real-world systems. As mentioned, our method is orthogonal to the choice of matching cost. Below, and in the supplementary materials, we also evaluate MC-CNN, the   state-of-the-art matching cost by Zbontar and LeCun [49], and show that it yields similar performance. For SGM's smoothness penalty (Eqn. 2) we use the following settings: P 1 = 100, P 2 = P 1 (1 + αe −|∆I|/β ), where α = 8, β = 10, and |∆I| is the absolute intensity difference at neighboring pixels. Our choice of P 2 favors large disparity jumps at high-contrast image edges.

Algorithm variants
In order to evaluate the full potential of SGM-P, we evaluate a number of different priors. We will substitute P with a combination of letters to distinguish the algorithm variants (see Table 1 for a summary and Fig. 4 for visualizations). The first letter distinguishes priors G derived from groundtruth disparities with priors E estimated from the input images. The former versions can be considered oracles that provide an upper bound on the potential benefit of our idea, while the latter versions give an indication of the actual realizable benefit. The second letter denotes the type of surfaces acting as priors: S for arbitrary (e.g., curved) surfaces, P for planar surfaces, and N for the case when only surface normals are available. Finally, we use i and v to distinguish between 2D priors only requiring an offset image, and 3D priors requiring an offset volume.
For the quantitative analysis we focus on only one E variant that estimates priors from the input images. We do this by running SGM at a coarser resolution and clustering the resulting disparities into disparity plane hypotheses [30]. We use these hypotheses and the associated pixel-to-plane label map to generate 2D and 3D orientation priors (EPi and EPv). For the 2D variant, SGM-EPi, we segment the image into superpixels [1] and then select for each superpixel the plane most often assigned to its constituent pixels. Superpixels with low support for any of the plane hypotheses are set to an arbitrary fronto-parallel plane.
The more powerful 3D variant, SGM-EPv, allows modeling of multiple disparity surface hypotheses at the same pixel. We use the same pixel-to-plane label map as for SGM-EPi but obtain potentially overlapping disparity hypotheses by bounding each 3D disparity plane by the convex hull of its constituent pixels in the label map.
For the oracle priors G based on ground-truth disparities we compare all three surface variants (S, P, and N) in order to explore the benefits and limitations of the different types of priors. Of these, SGM-GS uses the groundtruth disparity surface as 2D prior directly, which is the best possible prior available. Next, SGM-GP uses a piecewiseplanar approximation of the ground-truth surface, again as 2D prior, constructed in the same manner as SGM-EPi. We omit the suffix "i" in both cases since we do not have corresponding 3D priors. (While a 3D variant of SGM-GP with multiple overlapping planar hypotheses is possible, we found that it yielded no benefit over the 2D version.) Next, SGM-GNv discards the original ground-truth surface and uses only its normal map, which results in a 3D prior as explained in Section 3.5. We also include a "strawman" 2D version, SGM-GNi, which we obtain by integrating a single z-surface at an arbitrary depth. Finally, we investigate a 3D normal prior estimated from the images using a Manhattanworld assumption [32]; deviating from our naming scheme we simply call it SGM-MW (more details on this below).

Quantitative analysis
We start by evaluating the promise of SGM-P on a subset of the high-resolution stereo pairs from the training set of the Middlebury stereo evaluation v3 [38], for which highquality ground-truth disparities are available. We select 4 challenging image pairs with untextured slanted surfaces, depicted in Fig. 5. In the experiments below, we use the fullresolution (5-6 MP) versions of these datasets; see the supplementary materials for additional results, including other resolutions.
In our first experiment, we compare the baseline SGM method with our SGM-P algorithm using both estimated planar priors (SGM-EPi and SGM-EPv) and ground-truth priors (SGM-GP and SGM-GS). Fig. 6 shows disparity error rates (percentage of pixels whose disparity error is greater than t=2.0) as a function of completeness (inverse sparsity). We obtain disparity maps with increasing completeness by raising the allowable uncertainty U p (Eqn. 6) from 0 to U max . Our plots are similar to ROC curves and allow the comparison of sparse (or semi-dense) stereo methods that leave uncertain regions unmatched [29]. The error rate for the dense result (100% completeness) is the rightmost point on each curve.
The plots in Fig. 6 show that the four variants of our SGM-P algorithm all significantly outperform the baseline SGM algorithm on these four image pairs. As expected, the best performance is obtained with perfect orientation priors derived from the ground-truth surface (SGM-GS), which yields a dramatical improvement over SGM, with error rates ranging from one half to one fifth of the original errors. As mentioned, this provides an upper bound on the potential benefit of our idea. A more realistic upper bound is given by SGM-GP, which utilizes piecewise planar priors derived from the ground-truth disparities. This results in a slight decrease in performance compared to SGM-GS, but still a dramatical increase over the baseline. Most importantly, even without utilizing ground-truth information, we still get a significant improvement from planar priors estimated from the input images (SGM-EPi and SGM-EPv). The two versions, which are almost indistinguishable in terms of performance, achieve between 25% and 50% of the upper bounds, resulting in an improvement over the original SGM errors by 13-41%. For now we will focus on the simpler SGM-EPi method; we will discuss the potential of 3D priors (SGM-EPv) below. Fig. 7 shows the disparity maps and error maps for the dense results (100% completeness) for SGM, SGM-EPi, and SGM-GS.
It should be noted that the benefit of SGM-P strongly depends on the scene structure. In scenes with mostly frontoparallel surfaces, SGM-P yields little improvement. An important question is whether estimated priors can hurt the performance. Fig. 8 shows the performance of SGM-EPi and SGM-GS on all 15 Middlebury training pairs, sorted by rate of error reduction of SGM-EPi over SGM. It can be seen that the performance gains range from around -1% to 41%, with an average gain of 12%. Importantly, the performance never significantly decreases. We see the same trend for other matching costs; see the supplementary materials. We also submitted the results of SGM-EPi to the Middlebury stereo evaluation [39]. Since only a single submission per paper is allowed, we do not have a baseline for the 15 Middlebury test pairs. We thus cannot show the improvement ratios for the test pairs, as we do in Fig. 8 for the training pairs. The public evaluation results, however, show that our method outperforms all existing SGM entries in the public table (especially the full-resolution SGM entry). The largest performance gains are on scenes with slanted textureless surfaces, including Classroom, Crusade, and Stairs. SGM-EPi ranks 20th and 24th overall on test and training sets, respectively; among the full-resolution submissions it ranks 3rd on both sets. Table 2 shows the official (weighted) average training error rates, as well as the table ranks, for the different SGM-P variants for both NCC and MC-CNN [49] matching costs. While MC-CNN yields slightly lower errors, both costs result in similar performance gains. Recall that our goal is not to create a top-ranked stereo method, but rather to improve upon SGM, one of the most widelyused stereo methods. The rankings clearly show the potential of SGM-P for high-resolution stereo matching.
Finally, the fact that SGM-EPi and SGM-EPv produce very similar numerical results is not too surprising since most regions in the Middlebury images can be well explained with single surfaces. In the supplementary materials we show qualitative evidence that SGM-EPv is better at recovering surface creases by utilizing multiple overlapping hypotheses. Harnessing the full power of SGM-EPv, however, would require more powerful methods for generating priors that extend over larger regions of the image.

Manhattan-world priors
We exploit Manhattan-world layouts as a type of surface normal priors in our SGM-MW method. Using a method for scene layout recovery based on vanishing points [32], we obtain a semi-dense pixel labeling of the Manhattan world's principal surface normals in the left input image (Fig. 9b). We turn these normal priors into 3D disparity orientation priors by rendering planar segments in scene (z) space. Rather than covering the depth range uniformly, we select depths with local evidence for a surface, found by running SGM at a coarser resolution. Instead of fitting planes to these disparities (as we do for SGM-EPi) we convert the disparity map to z space, fit constrained planes with known orientation to the 3D points and convert those planes back to d space (see Section 3.5). Finally, these disparity planes are rasterized and used as 3D priors. Fig. 9 shows qualitative results of SGM, SGM-EPi, and SGM-MW on two challenging indoor image pairs. We see that both SGM and SGM-EPi struggle in regions with slanted untextured surfaces, where SGM-EPi does not find a supporting plane. However, SGM-MW recovers smoother slanted surfaces in these regions by utilizing the Manhattanworld normal estimates.
In the supplementary materials we also test our surface normal prior idea with oracle priors derived from groundtruth data. Recall that a planar surface patch with known normal n but unknown depth z yields a family of disparity planes whose orientation depends on z, which requires a 3D prior (SGM-GNv). We show that modeling this depth dependance is crucial by demonstrating that SGM-GNv is significantly more accurate than the "strawman" 2D version SGM-GNi, which integrates the surface normals in scene space at one arbitrary depth and uses the resulting disparity surface as a 2D prior. Middlebury training pairs. All timings were measured on a computer with a 3.4 GHz Xeon E5-2643 v4 processor and 32 GB RAM. Our C++ implementation is not yet optimized for speed. However, the timings show that SGM-GS has almost no overhead over SGM. SGM-EPi and SGM-EPv have similar runtimes, with average runtime overheads over SGM of about 7%. This overhead is mainly due to the cost of extracting the orientation priors.

Conclusion
We have presented a simple extension to semi-global matching (SGM) that allows surface orientation priors to be incorporated as soft constraints. Using priors derived from stereo matching at coarser resolution, our SGM-P method consistently yields improved accuracy for challenging indoor scenes that contain slanted weakly-textured surfaces. We also demonstrate the potential of orientation priors derived from single images. Our analysis involving oracle priors demonstrates the potential for large performance gains.
Avenues for future work includes recovering more accurate orientation priors, possibly via semantic analysis [2] or revisiting binocular photometric stereo [13]. Combining orientation priors with depth priors, obtained either from coarse resolution or via commodity depth sensors, is also worth exploring. Finally, it might be possible to extend our method to other MRF optimization frameworks with firstorder smoothness terms.

A. Other matching costs and resolutions
In addition to NCC we also evaluate MC-CNN [49] as a matching cost, on both full-resolution (6MP) and halfresolution (1.5MP) versions of the Middlebury training images. Fig. A1 shows barplots for baseline SGM, estimated priors SGM-EPi, and ground-truth priors SGM-GS, on all four combinations of matching cost and image resolution. Table A1 summarizes the average performance gains.
For SGM-EPi, which estimates plane hypotheses from coarser matching results, we run SGM with NCC matching costs at quarter resolution for all of these combinations. We found that MC-CNN is tuned for half resolution and does not work well at quarter resolution. Fig. A1   Despite the low quality of the estimated planes, the priors result in significantly cleaner disparity maps compared to the baseline. In addition, SGM-EPv yields cleaner surface transitions ("creases") in the presence of noisy labels and/or competing priors, for instance near the front edge of the right corridor wall (highlighted).

B. Oracle surface normal priors
To allow an accurate comparison of the achievable benefit of priors derived from surface normals (as opposed to the surfaces directly) we include additional experiments using oracle priors derived from ground-truth data.
In order to compute oracle priors from ground-truth surface normals (SGM-GNv), we integrate these normals in scene space under a perspective projection model using a least-squares approach [45]. The result is a z-surface in world coordinates, initially at an arbitrary depth. We implement the 2D integration step using a sparse solver employing conjugate gradients. We divide the image into a coarse grid and independently integrate a surface in each grid cell, arbitrarily fixing one depth value in each cell. In order for the integration to succeed, it is crucial that the discontinuities in the normal map are known. Otherwise any integration method, including our least-squares approach, will not produce a locally accurate z-surface.
Recall from Section 3.5 in the paper that a planar surface patch with known normal but unknown depth yields a family of planar disparity surfaces whose orientation depends on z, resulting in a 3D prior SGM-GNv. To investigate the importance of modeling this depth dependance, we compare the accurate 3D version SGM-GNv with an (inaccurate) 2D version SGM-GNi which we obtain by integrating the surface normals in scene space using an arbitrary starting depth, and using the resulting disparity surface as a 2D prior. We compare both the accurate 3D version SGM-GNv and the 2D "strawman" SGM-GNi with the ground-truth surface prior SGM-GS and its piecewise-planar approximation SGM-GP. Fig. A2 shows the performance of these variants. It can be observed that the accurate normal prior SMG-GNv is close to the upper bound SGM-GS, often outperforming the planar approximation SGM-GP, while the strawman SGM-GNi performs much worse. The difference between SGM-GNv and SGM-GNi is less pronounced on other image pairs where fewer slanted surfaces are present, or where the single integration result coincides with a large actual surface in the scene.

C. Estimated 2D vs. 3D priors
Recall from Table 2 in the paper that estimated 3D priors (SGM-EPv) perform quantitatively slightly better than 2D priors (SGM-EPi). Fig. A3 shows a qualitative example illustrating why 3D priors are advantageous. It can be seen that both SGM-EP methods clearly produce much cleaner surfaces than the baseline SGM algorithm. In addition, SGM-EPv produces smoother results especially near plane transitions at discontinuities and orientation changes ("creases") between planes. Since SGM-EPi only has a single orientation prior per pixel, its performance degrades when the pixel-to-plane labeling is noisy or incomplete. SGM-EPv allows multiple overlapping disparity hypotheses and is thus more robust in the presence of noisy labels.