{'fraction_non_alphanumeric': 0.028076823701662215, 'fraction_numerical': 0.0075790198949272245, 'mean_word_length': 4.802848575712144, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Object 3D Reconstruction based on Photometric Stereo and Inverted Rendering

Methods for 3D reconstruction such as Photometric stereo recover the shape and reflectance properties using multiple images of an object taken with variable lighting conditions from a fixed viewpoint. Photometric stereo assumes that a scene is illuminated only directly by the illumination source. As result, indirect illumination effects due to inter-reflections introduce strong biases in the recovered shape. Our suggested approach is to recover scene properties in the presence of indirect illumination. To this end, we proposed an iterative PS method combined with a reverted Monte-Carlo ray tracing algorithm to overcome the inter-reflection effects aiming to separate the direct and indirect lighting. This approach iteratively reconstructs a surface considering both the environment around the object and its concavities. We demonstrate and evaluate our approach using three datasets and the overall results illustrate improvement over the classic PS approaches.


I. INTRODUCTION
Scene and object 3D reconstruction is the process of capturing their shape and appearance using various methods and approaches such as stereo, structure from motion, shape from shading, and many more [35]. The reconstruction is highly applicable in a number of fields as it provides the ability to understand 3D scenes and objects on basis of 2D images. The applications ranging from robotics and automated industrial quality inspection over human-machine interaction [6] (example action, gesture and face recognition), satellite 3D data analysis [25], to movies and architectural applications [17]. Additionally, the method is commonly used to analyse the surfaces of a celestial object, such as the Moon [18].
Photometric stereo (PS) is a well-established technique that is used for 3D surface reconstruction [10]. The approach generally inherits the principle of appearance analysis of a 3D object on its 2D images. Based on the intensity information, these approaches attempt to infer the shape of the depicted object [17]. It estimates shape and recovers surface normals of a scene by utilising several intensity images obtained under varying lighting conditions with an identical viewpoint [16], [41]. By default, PS assumes a Lambertian surface reflectance; a standard reflectance model which defines a linear dependency between the normal vectors and image intensities. The definition of the model then can be used to determine the 3D space in the image [5]. However, just a single Lambertian image is not adequate to correctly determine the surface shape. Therefore, the PS uses several images whose pixels corresponds to a single point on the object and is able to recover surface normals and albedos [40].
Light displays complicated attributes while interacting with objects resulting direct and indirect illumination as shown in figure 1.However, classical PS naively assumes that a scene is illuminated only directly by the emitting source. In presence of indirect illumination, it produces erroneous results with reduced reconstruction accuracy [21]. For example, an indirect illumination such as inter-reflections makes concave objects appear shallower [30].
In this paper, we present an iterative 3D reconstruction method considering inter-reflections due to the concavities and the environment. We propose a novel method that accounts for inter-reflections in a calibrated photometric stereo environment. This approach utilises a reverted Monte Carlo ray tracing method to extract the environmental colour trying to minimise the inter-reflections within images used for photometric stereo. This approach not only accommodates the concave surface but also applies to any object in a scene with inter-reflections. The proposed method Iterative Ray Tracing Photometric Stereo -IRT PS iteratively applies Photometric Stereo (PS) and a reverted ray tracing algorithm based on a Monte-Carlo implementation to reconstruct with higher accuracy the observed surfaces. This approach iteratively reconstructs the surface and separates the indirect from direct lighting considering also the environment around the object. Likewise, the proposed IPT-PS method can be integrated to any PS technique removing the effects of inter-reflections and improving the overall reconstruction accuracy.
Our approach is extensively evaluated on three datasets and the overall results demonstrate improvement over the classic approaches. The main contributions of our work are: • an iterative surface reconstruction method that is utilised by the reverted Monte Carlo ray tracing • the proposed methodology that allows IRT-PS to be combined with any other PS algorithm improving the overall performance.
The paper is organised as follows: Section II provides background material on Photometric Stereo, followed by invert light transport, their properties and related works. In section 3, we introduce the mathematical definition of necessary the terms. In section 4, we propose a novel iterative PS method and discuss the suggested reverted Monte Carlo ray tracing algorithm. The performance of this approach is investigated in section 5, with section 6 concluding the work.

II. PHOTOMETRIC STEREO
Photometric stereo (PS) is an approach to estimate the surface normal and reflectance (i.e albedo) of an object based on three or more intensity images with the fixed view under varying lighting condition [16]. A number of solutions have been proposed to address this problem. Woodham [45] was the first to introduce the PS method. He proposed an approach simple and effective. However, he only considered Lambertian surface which suffers from noise. In his method, it is assumed that the surface albedo is known a prior for each point on the surface, the surface gradient can be obtained by using a threepoint light source. Onn and Bruckstein [33] developed a twoimage PS method. Their work was based on the assumption that the objects are smooth and no self-shadows are present. The PS was further extended by Coleman and Jain [9], which utilises four light sources, discards the specular reflections and estimates the surface shape by performing mean of diffuse reflections and the use of the Lambertian reflection model. Nayar et al. [29] proposed a PS method which used a linear combination of an impulse specular component and the Lambertian model to recover the shape and reflectance for a surface. Similarly, an algorithm for estimating the local surface gradient and real albedo from four sources in the presence of highlights and shadows was proposed by Barsky and Petrou [3]. Chandraker et al. [7] proposed an algorithm that required at least four light sources and images to reconstruct surface in presence of shadow. It is also worth mentioning the related work presented in [1], [11], [28], [34] following similar architectures and approaches.
Furthermore, over the previous years, methods that consider images produced by more general lighting conditions not known a prior. Basri et al. [4] proposed a PS method where no prior knowledge of the light source and its type is required, however the emitting source should be distant or unconstrained. They utilised low order spherical harmonics and optimised it to low-dimensional space to represent Lambertian objects. Likewise, Shi et al. [36] used colour and intensity profiles, which are obtained from registered pixels across images to propose a self-calibrating PS method. They automatically determine a radiometric response function and resolved the generalised bas-relief for estimating surface normals and albedos. While lighting conditions could be unknown, they required fixed viewpoint.
Nevertheless, a majority of the methods and models while working well with with matte objects, under-perform when the reconstructed objects are specular, transparent or with interreflections. Non-Lambertian reflection and specifically interreflection may be difficult to solve in photometric stereo. Solomon and Ikeuchi [37] developed a method where they utilised four lights and tried to extract the surface shape and roughness of an object which has specular lobe. They used a simplified version of Torrance-Sparrow reflectance model to determine the surface roughness. Bajcsy et al. [2] presented an algorithm for detecting diffuse and specular interface reflections and some inter-reflections. They used brightness, hue, and saturation values instead of RGB as they point out that the values have a direct correspondence to body colours and to diffuse and specular, shading, shadows and interreflections. But, the algorithm requires uniformly coloured dielectric surface under single coloured scene illumination. Tozz et al. [43] proposed a PS method that is independent of the albedo values and uses image ratio formulation. However, their method requires an initial separation of diffuse and specular components.
In addition, because of the nature of light, inter-reflection is unavoidable even in a controlled environment. This may vary in magnitude depending on the environment itself, the structure, and the material of the object. Moreover, it may not be uniform over the whole surface. As a result, the images are blurred locally in shade. Most photometric methods do not consider inter-reflection from an environment and concave surfaces, and those that do have considered one of the two cues only. One of the first attempts at scene recovery under interreflection was purpose by Nayar et al. [30]. They presented an iterative algorithm which recovers shapes from a concave surface which first estimates the shape from intensity data; then this shape is used as input, and the radiosity method is applied to estimate a corrected, no-interreflection image intensity distribution. These steps are carried iteratively until convergence. Nevertheless, the algorithm only examines the inter-reflection in concave shapes, Lambertian reflectance models and does not take into account the colour of the interreflected light. Funt and Drew [14] proposed an algorithm which is based on singular value decomposition of the colour for a convex surface. They proposed a "one-bounce" model which measured inter-reflection between two matte convex surfaces with a uniform colour and illumination can vary spatially in its intensity but not in its spectral composition. Again, the algorithm is specific to convex surface assuming a uniform colour and illumination that can vary spatially. Langer [27] did the study on the shadows which becomes inter-reflections. They proposed a method for inferring surface colour in a uni-chromatic scene which is based on the relative contrast of the scene in different colour channels. Again, the method is highly specific and only deals with inter-reflection related to shadow.
Most existing shape from intensity techniques accounts for an only direct component of light transport. Nayar et al. [31] proposed using high-frequency illumination patterns to separate direct and indirect illumination from more general scenes.Gupta et al. [15] studied the relation between illumination defocus and global light transport. Again, Chen et al. [8] used modulated structured light patterns with high-frequency patterns to mitigate the effects of indirect illumination. Lamond et al. [26] used high-frequency light patterns to separate the diffuse and specular components of BRDF. Holroyd et al. [19] constructed a high-accuracy imaging system for measuring the surface shape and BRDF. All these techniques either are an active method or they assume that the indirect illumination in each of the acquired images is caused by a single source. In contrast, we consider separation of indirect components by simulating the inter-reflections and removing it from the source images.

III. FORWARD LIGHT PROPAGATION
An image captured by the camera is the results of a complex sequence of reflections and inter-reflections. When light is emitted from the source, it bounces off the scene's surface one or more times before reaching to a camera. In theory, every image can be captured as infinity sum, I = I 1 + I 2 + I 3 + .. + I n , where I n denotes the total contribution of light that bounces n times before reaching the camera as shown in figure 1. For example, I 1 is the captured image if it was possible to remove all the indirect illumination from reaching the camera sensor, while the infinite sum I 2 + I 3 + ..+I n describes the total contribution of indirect illumination. Although we can capture the final image I using a camera, the individual n-bounce images are not directly measurable in the real-world scenario.
Nevertheless, the techniques for simulating inter-reflections and other light transport effects are not new in the computer vision and graphics. The algorithm that simulated the forward light transport was solved by Kajiya [24]. The algorithm is also known as rendering equation. The rendering equation is an integral in which the radiance leaving a point is given as the sum of emitted plus reflected radiance under a geometric optics approximation.
(1) Where I(x, x ) is related to the intensity of light passing from x to point x. g(x, x ) is a "geometry" term, e(x, x ) is related to the intensity of emitted light from x to x and p(x, x x ) is related to the intensity of light scattered from x to x by a patch of surface at x .
An algorithm such as ray tracing [12] [23] solved the equation 1 by using Monte-Carlo methods, whereas radiosity [12] [22] used finite element method to produce near realistic looking images in the field.
For a Lambertian object illuminated by a light source of parallel rays, the observed image intensity a at each pixel is given by the product of the albedo ρ and the cosine of the incidence angle θ i (the angle between the direction of the incident light and the surface normal) [20]. The above incidence angle can be expressed as the dot product of two unit vectors, the light direction l and the surface normal n, a = ρ cos(θ i ) = ρ(l · n).
Let us now consider a Lambertian surface patch with albedo ρ and normal n, illuminated in turn by several fixed and known illumination sources with directions l 1 , l 2 , ..., lQ. In this case we can express the intensities of the obtained pixels as: We stack the pixel intensities to obtain the pixel intensity vector A a = (a 1 , a 2 , ..., aQ) T . Also the illumination vectors are stacked row-wise to form the illumination matrix L = (l 1 , l 2 , ..., lQ) T . Equation (2) could then be rewritten in matrix form: If there are at least three illumination vectors which are not coplanar, we can calculate ρ and n using the Least Squares Error technique, which consists of using the transpose of L, given that L is not a square matrix: Since n has unit length, we can estimate both the surface normal (as the direction of the obtained vector) and the albedo (as its length). Extra images allow one to recover the surface parameters more robustly.

IV. PROPOSED ITERATIVE RAY TRACING PHOTOMETRIC STEREO METHOD (IRT-PS)
In nature, when we illuminate a surface, light not only reflects towards the viewer but also among all surfaces in the environment. This is always true, with exception of scenes that consists only of a single convex surface. In general, scenes include concave surfaces where points reflect light between themselves. Furthermore, inter-reflections can occur due to the environment and appreciably can alter a scene's appearance. In figure 2, to simulate the inter-reflections the sphere is placed within the Cornell box [32] and highlights the inter-reflections i.e sphere receive the colours from its environment. Existing computer vision algorithms do not account for effects of inter-reflections and hence often produce erroneous results. The algorithms that are directly affected by interreflections are the shape-from-intensity algorithms including Photometric Stereo. Due to the common assumption of single surface reflections (direct illumination) and disregarding higher order (inter-reflections, a subset of global illumination), photometric methods produce erroneous results when applied to open scenes.
The first stage of this approach (stage 0), is performed only once throughout the process and involves the acquisition of the initial input images. It is assumed that inter-reflections are present and that the captured surface is within the known environment. In our case within a Cornell Box.
Moving to the following stage, PS is applied to the images acquired at stage 0 using equation 4 to obtain the initial albedo ρ t and normals n t . Integrating over the obtained normals a 3D surface H t is obtained using the M-estimator technique. This initial surface that is affected by the presence of the inter-reflections becomes the input to the following stage, that involves the proposed reverted ray tracing algorithm.
As environment information is known prior to reconstruction, we can implement our environment. The Cornell Box was setup as the environment at the following stage 3. More realistic textures can be used for the walls without affecting the proposed methodology.
In stage 4, we simulate the environment assuming the Cornell box is given or estimated. In our case, this approach can be extended to other realistic environmental projection such as Hemispherical Dome Projection [39] without affecting the proposed methodology. Then we place the generated H t surface within this environment.
In the following stage, based on equation 7, the reverted ray tracing algorithm is applied. Since we are only interested in inter-reflections, only the indirect illumination is calculated.To implement the ray tracer for Lambertian surface, we solve the rendering equation by integrating Monte Carlo estimator Where L 0 (p, w 0 ) is the total outgoing radiance reflected at p along the w 0 direction. L i (p, w i ) is the radiance incident at p along the w i direction. f (p, w 0 , w i ) determines how much radiance is reflected at p in direction w 0 , due to irradiance incident at p along the w i direction. cosθ i is from the Lambert's cosine law: diffuse reflection is directly proportional to cos(θ) is an integral over a given hemisphere.
As Monte-Carlo approximation is a method to approximate the expectation of a random variable, using samples.
where, E(X) is an approximation of average value of random variable X.N is the sample size. And when we integrate it to equation 5 we solve the rendering equation.
However, Monte-Carlo estimator is affected by noise, the ray tracer algorithm also inherited such a problem. For example, to half the noise in an image rendered by ray tracing, we need to quadruple the number of samples.
To estimate the environmental colour, we first hit the H t surface with rays from each pixel, consider techniques such as hemisphere sampling and we randomly reflect the rays toward the environment. As a result, the images of the environment are captured for the various levels/depths of ray reflection. In this study, we only use up to 3 reflection rays (1 to 3) with just a single sampling, as shown in figure 5. Because we are not calculating all the ray reflections within the environment, we will have pixel locations without intensity values. An example can be seen in figure 6. Therefore, we are using a non-uniform interpolation algorithm [42] to approximate the missing values in the obtained environmental intensity images E r t , where r corresponds to the number of ray reflections.
In figure 6, we see that the more ray reflects, the less bright the pixels become. The main reason behind this phenomenon is because of ray tracing algorithm and considering that the first ray r1 has more influence on the final pixel intensity than the ray r3. Therefore, when we have more ray reflections, the intensity of the pixels needs to be reduced, accordingly.
In stage 5, we generate the new input images A t+1 = A t − E r t by subtracting the environmental intensity reducing the inter-reflections from the original input images. There are three different sets of images for each ray reflection r1, r2 and r3.
Finally, the obtained images which have fewer interreflections (example difference image is shown in figure 7) are used for as input to photometric stereo, generating a new H t+1 surface. The whole process can be applied iteratively for a certain number of iterations or until the difference D H = H t+1 − H t between a new 3D surface and the previous one is less than a given threshold.

V. EXPERIMENTS AND RESULTS
In our comparative evaluation study, three different datasets with ground truth were used. Scan data from the Harvard PS dataset [13], a dataset with faces [44] and synthetic data generated by simulated objects (see figures 8 and 11).
We used the photometric stereo approach to reconstruct the sets of the acquired H t surface, with and without interreflections considering different numbers (1 to 3) of ray reflections in the proposed reverted Monte-Carlo ray tracing algorithm. We then estimate the height-, albedo-and normal-   error comparing to classic PS method [38] using the available ground truth.
To calculate the height-error we used the equation, H err is the mean for height error. H GT is the height value of ground truth surface, whereas H t is the height value of reconstructed surface. Regarding the albedo-error we use the equation below, where P rgb err is the albedo-error from mean of individual colour channel; Red P r err , Green P g err , and Blue P b err channel. Likewise, to calculate normal-error we utilise the following equation: N xyz err denote the mean normal-error for all the axis x, y, and z.Where N x err is a mean error for X axis, N y err is mean error for Y, and N z err is mean error for Z, N xyz H is normal from reconstructed surface. Fig. 10: Example of the estimated albedo using classic PS [38], and the proposed IRT-PS method using 1-, 2-and 3-ray reflections.
From the table 1, and charts in figure 11, we can see that the overall trend of mean Height, Albedo, and Normal errors are reduced with our approach than the classic photometric stereo one. In table 1, text highlighted in red are the average overall results of the [38] photometric stereo method. Whereas best results from our IRT-PS approach are highlighted in the green text. From the charts figure 11, we can see the general trend of the height error: Results improve with each additional ray and the best result is achieved by Ray 3. Likewise, the best result for Albedo and Normal are given by Ray 2. The indirect illumination captured by Rays R3 and R2 of the

VI. CONCLUSIONS
In this work, a novel iterative method considering interreflections both due to concavities and the environment was proposed. The IRT-PS approach iteratively applies Photometric Stereo and a reverted Monte-Carlo ray tracing algorithm, reconstructing the observed surface and separating the indirect from direct lighting. A comparative study was performed evaluating the reconstruction accuracy of the proposed solution on three different datasets and the overall results demonstrate improvement over the classic approaches that do not consider environmental inter-reflections.