{'fraction_non_alphanumeric': 0.037906525953047596, 'fraction_numerical': 0.012635508651015866, 'mean_word_length': 4.804166666666666, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust Surface Reconstruction from Gradients via Adaptive Dictionary Regularization

This paper introduces a novel approach to robust surface reconstruction from photometric stereo normal vector maps that is particularly well-suited for reconstructing surfaces from noisy gradients. Specifically, we propose an adaptive dictionary learning based approach that attempts to simultaneously integrate the gradient fields while sparsely representing the spatial patches of the reconstructed surface in an adaptive dictionary domain. We show that our formulation learns the underlying structure of the surface, effectively acting as an adaptive regularizer that enforces a smoothness constraint on the reconstructed surface. Our method is general and may be coupled with many existing approaches in the literature to improve the integrity of the reconstructed surfaces. We demonstrate the performance of our method on synthetic data as well as real photometric stereo data and evaluate its robustness to noise.


INTRODUCTION
Imaging techniques such as photometric stereo [1] allow one to efficiently estimate the normal vector map of an object. The primary goal of such methods is to ultimately derive a threedimensional representation of the object, which requires some flavor of numerical integration of the gradient fields defined by the normal vector map. Robust photometric stereo-the problem of accurately determining the normal map of a nonideal surface or from noisy data-has attracted considerable attention in recent years [2,3,4]. In this work, we seek to develop a robust approach to the problem of reconstructing surfaces from gradient fields that can accurately estimate a 3D representation of an object in the presence of noise.
The problem of reconstructing a surface from estimates of its photometric stereo gradient fields has been investigated since the late 1980s. The seminal works of Simchony et al. [5] and Frankot and Chellappa [6] seek to solve the problem through a least squares approach, utilizing efficient discrete This work was supported in part by the following grants: ONR grant N00014-15-1-2141, DARPA Young Faculty Award D14AP00086, and ARO MURI grants W911NF-11-1-0391 and 2015-05174-05.
Fourier transform (DFT) or discrete cosine transform (DCT) based solvers. Harker and O'Leary [7] propose a modified "global" least squares problem and extend this method to incorporate regularization [8], solving a Sylvester equation to obtain the solution. Recently, Quéau and Durou [9] introduced a weighted-least squares formulation as well as formulations minimizing total-variation and incorporating the 1 norm to promote sparsity. Further attempts at applying a regularization term while integrating the gradients have also been proposed at the expense of computation time [10,11]. Additional approaches include line-integral based methods [12,13] and reconstructions based on the calculus of variations [14,15,16]. A range of other methods have also been proposed with mixed results [17,18,19,20,21,22].
Our work builds on these previous works, specifically those that utilize a least squares-type formulation to relate the underlying surface and its gradient fields. In particular, we propose a novel adaptive dictionary learning framework that enables the robust estimation of surfaces from noisy gradients. Dictionary learning [23,24,25] has, in recent years, been successfully applied to many imaging applications, e.g., [26,27,28]. In dictionary learning models, one typically seeks to learn sparse representations of local regions (patches) of the data. These models often induce a type of smoothness constraint on the underlying data that, in the case of surface reconstruction, we show leads to robust reconstructions with desirable noise rejection properties. Our framework is general and can be easily combined with any existing method that utilizes a least squares-type objective to estimate the underlying surface.

SURFACE RECONSTRUCTION FROM GRADIENT FIELDS
Let n(x, y) ∈ R 3 denote the normal vector of a differentiable surface z(x, y) at position (x, y), and let n 1 (x, y), n 2 (x, y), and n 3 (x, y) denote the x, y, and z components of this vector, respectively. Under this ideal model, one can relate the x and y derivatives of the surface z to its normal vectors via the relation where we have defined p(x, y) := n 1 (x, y)/n 3 (x, y) and q(x, y) := n 2 (x, y)/n 3 (x, y). In practice, the estimated (e.g., by photometric stereo) normal vectors of a surface and its gradient fields will not exactly satisfy (1), so one must instead find a function z(x, y) with derivatives close to p(x, y) and q(x, y) in an appropriate sense, often by minimizing a variational problem of the form (2) When our data is instead sampled on a discrete grid, we will not have access to a continuous normal map n(x, y) but will instead have a matrix N ∈ R m×n×3 containing the normal vectors of the object on the grid. Following (1), we can compute matrices P ∈ R m×n and Q ∈ R m×n containing the measured gradients, and our goal then becomes to estimate the matrix Z ∈ R m×n containing the values of the surface z(x, y) sampled on the grid. The discrete analogue of (2) is commonly expressed [5,6,7] as a standard least squares problem of the form where z = vec(Z) ∈ R mn is the vectorized surface, A is a numerical differentiation operator, and the vector v is an appropriate function of the measured gradients, P and Q. Solving this problem yields a representation of our surface that is optimal in the least squares sense. Note that the specific forms of A and v can vary. One possible formulation is where D n is the discrete first differences matrix, and ⊗ denotes the Kronecker product. However, multiple models are possible, each yielding reconstructed surfaces with different properties. Importantly, the dictionary learning based approach that we introduce in Section 3 can be coupled with any least squares model of the form (3), so our proposed approach is quite flexible.

ADAPTIVE DICTIONARY LEARNING REGULARIZATION
Given normal vectors corrupted by noise or other nonidealities, solving (3) directly generally produces a rough, bumpy surface, even when the underlying true surface is smooth. Thus, in this work, we propose an adaptive dictionary regularizer that can be combined with the least squares model (3) to more accurately estimate the underlying surface.
In particular, we propose to solve the dictionary learning problem arg min Here, P j is a patch extraction operator that extracts a vectorized c x ×c y spatial patch from z, D ∈ R cxcy×K is a dictionary matrix whose columns d i are vectorized c x × c y atoms, and B ∈ R K×c is a matrix of sparse codes, where column b j of B defines the (usually sparse) linear combination of atoms used to represent the patch P j z of z. Also, . 0 is the familiar 0 "norm" and λ, µ > 0 are regularization parameters. We include the constraints b j ∞ ≤ a, where a is typically very large, to prevent pathologies that could theoretically arise during minimization since (5) is non-coercive with respect to B, but the constraint is inactive in practice [29]. In addition, we constrain the columns of D to be unit-norm to avoid scaling ambiguity between D and B [30]. Note that (5) is a non-convex problem.
By solving (5), we are attempting to estimate our surface z by numerically integrating its gradient fields through a least squares functional while simultaneously enforcing that local patches of the reconstructed surface should have sparse representations with respect to the dictionary D. As D itself is learned, our proposed algorithm can automatically adapt to the underlying properties of the surface and its gradients. Intuitively, since the same dictionary of atoms is used to (sparsely) represent all patches, the atoms can learn universal features of the surface. This effectively equips (5) with global information to estimate each patch, which (as we will show) can yield robust reconstructions when the gradients are noisy/corrupted.

Dictionary Learning on Surfaces (DLS) Algorithm
We propose to solve (5) via a block coordinate descent-type algorithm where we alternate between updating z with (D, B) fixed and updating (D, B) with z fixed. Henceforward, we refer to this algorithm as the Dictionary Learning on Surfaces (DLS) method.

(D, B) updates
Let P be the matrix with columns P j z. With z fixed, the minimization of (5) with respect to (D, B) is We solve (6) via a block coordinate descent method where we iteratively minimize the cost with respect to the ith row of B and the ith atom, d i , of D for every 1 ≤ i ≤ K with all other variables held fixed. A full derivation of this step can be found in [29,31], which we omit here due to space considerations.

z update
With D and B fixed, our problem becomes The cost in (7) can be written in the form f (z) + g(z), where We utilize a proximal gradient [32] strategy to solve (7), iteratively updating z according to where prox τ g (y) := arg min Definingz k := z k − τ ∇f (z k ), we see that (9) and (10) imply that z k+1 satisfies the normal equation The matrix on the left hand side of (11) is diagonal, so its inverse can be cheaply computed to solve for z k+1 . Note that (7) is a simple least squares problem and, as such, could be minimized by other methods (e.g., conjugate gradients).

RESULTS
In this section, we numerically evaluate our proposed DLS method on several datasets. In each case we compare our method to the spectral regularization method (SR) [8], the isotropic total variation (TV) method [9], and DCT based least squares (DCTLS) [5]. For methods that include tunable parameters, we sweep over a wide range of values, reporting the best results obtained. Our proposed DLS method can incorporate any least squares based solver by simply defining A and v in (5) accordingly. For all results given here, we use the least squares cost found in [5]. To evaluate the robustness of each algorithm, we add Gaussian noise to the data. For our proposed DLS method, we use dictionary atoms of size 8 × 8 pixels and a square 64 × 64 dictionary D, initialized with a DCT matrix. We extracted patches from z using a spatial stride of two pixels in each direction, allowing adjacent patches to overlap. Finally, we initialized z as the vectorized surface produced by solving the stand-alone least squares problem in [5], and initialized B = 0.

Synthetic Surface Reconstructions
To quantitatively evaluate our method, we first considered two synthetic datasets, which we call "Tent" and "Vase", for which we have analytic expressions for z = f (x, y). Given f (x, y), we can differentiate to obtain the gradients, ∂f (x, y)/∂x and ∂f (x, y)/∂y, and sample on a discrete grid. After reconstructing the surface from these gradients subject to additive noise, we evaluate the integrity of the reconstructions against the ground truth, f (x, y), via the SSIM metric [34]. In these experiments, we add noise directly to the gradient fields to achieve a prescribed signal-to-noise ratio (SNR). Figures 1 and 2 show the reconstructed surfaces produced by each algorithm. As these images illustrate, the proposed DLS method produces significantly smoother surfaces from noisy data compared to the existing methods. Intuitively, the locally sparse model imposed by the dictionary regularization denoises the surfaces, while the adaptive nature of the dictionary allows DLS to represent and reconstruct both sharp edges and smooth regions on a data-dependent basis. The surfaces obtained by SR, TV, and DCTLS are much more sensitive to the noisy gradients. Indeed, while they retain the general shape of the surface, they exhibit significantly more corruption. In particular, the spectral regularization method seems to introduce a systematic "rippling" into the reconstructions.  Tables 1 and 2 numerically corroborate the qualitative results from Figures 1 and 2. In the low SNR regime, DLS significantly outperforms the other approaches. As SNR increases, the gap decreases. When the data is essentially noiseless, DLS, TV, and DCTLS are all able to reconstruct the surfaces with comparably negligible error.

Photometric Stereo
We now return to the problem of reconstructing a 3D representation of an object from normal vectors obtained through photometric stereo. We consider a dataset containing 10 images, each taken under a unique, known lighting direction, and corrupt the images with Gaussian noise at a prescribed SNR. We compute the normal vectors from the noisy images using the standard least squares approach [2]. Given the normal vectors, we then compute gradient fields as discussed in Section 2, and from those we generate a 3D reconstruction of the object using each method. Figure 3 illusrates the results of this procedure on the Frog dataset, 1 which contains real images of a frog statue [33].
The reconstructions in Figure 3 showcase the ability of the proposed DLS approach to produce a smoother surface from the noisy gradients compared to the existing methods. The denoising capability of DLS may prove valuable when running photometric stereo on real-world data, where noise and other non-idealities are inevitable.

CONCLUSION
In this work, we explored the use of adaptive dictionary learning based regularization for the estimation of surfaces from their gradient fields. We showed that our proposed dictionary learning approach is able to effectively reject the addition of noise to gradient fields/images and produce more accurate and smooth representations of the underlying surfaces compared to existing methods. Our dictionary learning framework is very general and would be straightforward to combine with many existing algorithms. In future work, we plan to investigate these combinations and perform a more thorough study of the influence of the various parameters of our dictionary learning model on the reconstructed surfaces.