{'fraction_non_alphanumeric': 0.025824175824175823, 'fraction_numerical': 0.007348901098901099, 'mean_word_length': 4.7281667977970105, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Microstructure Surface Reconstruction from SEM Images: An Alternative to Digital Image Correlation (DIC)

We reconstruct a 3D model of the surface of a material undergoing fatigue testing and experiencing cracking. Specifically we reconstruct the surface depth (out of plane intrusions and extrusions) and lateral (in-plane) motion from multiple views of the sample at the end of the experiment, combined with a reverse optical flow propagation backwards in time that utilizes interim single view images. These measurements can be mapped to a material strain tensor which helps to understand material life and predict failure. This approach offers an alternative to the commonly used Digital Image Correlation (DIC) technique which relies on tracking a speckle pattern applied to the material surface. DIC only produces in-plane (2D) measurements whereas our approach is 3D and non-invasive (requires no pattern being applied to the material).


Introduction
We study 3D micsrostructure surface modeling over time for a material undergoing fatigue testing. Fatigue refers to the initiation and propagation of cracks in a material due to cyclic loading. The experimental study of fatigue at the microstructure level is often done through visual inspection of test materials using scanning electron microscopes (SEMs).
The scanning electron microscope (SEM) is the most commonly used tool for imaging of microstructures . These are structures with dimensions around 1 micron (10 −6 m). The image formed is a projected one, similar to traditional optical cameras. Figure (1) from [1] shows an example of such a study. A small rectangular block of nickel (roughly 10 µm × 30 µm) is placed under the SEM and repeatedly subjected to cycles of tension and compression in the lengthwise direction (y direction in the image). Over time cracks begin to form at internal planes at certain known orientations that relate to the crystal structure of the material.
The images show the growth of these cracks in both number and size. It is of interest to material scientists to under- Figure 1. Fatigue experiment conducted on a material with successive SEM images generated at progressive parts of the experiment showing the formation and propagation of cracks (image from [1]). stand the deformation of the surface which can be mapped to a strain tensor. Material Scientists then use the strain tensor to predict material life and failure modes. One technique commonly used today is Digital Image Correlation [3]. In this technique, shown in figure (2), a known socalled 'speckle' pattern is imprinted on the sample being tested and images are correlated over time to estimate how the surface is deforming. This technique, though, is invasive and limited to 2D estimates of the surface deformation.
We present an alternative approach that estimates the magnitude and shape of the extrusions and intrusions on the surface -in other words the depths and heights of the peaks and valleys relative to the surface, in addition to the lateral Cameras are used to record changes in a 'speckle' pattern that is imprinted on the sample being tested.
(Images taken from https://digitalimagecorrelation.org/) (in-plane) deformation. The surface extrusions/intrusions can more accurately inform the correlation between fatigue life and slip localization. Recent studies suggest that there is a direct correlation between those and that fatigue life can be predicted from these out-of-plane displacements. Our goal is to detect variations in the z-direction on the order of 0.1 µm.
During the experiment, the SEM cannot be re-oriented so all the images obtained are from a fixed head-on viewpoint. Once the experiment is complete, though, the SEM can be re-oriented to obtain additional views.
Our approach is summarized as follows. First we use stereo metrology to estimate the surface roughness at the end of the experiment using two images of the sample at the end of the experiment from different camera orientations. Then we use optical flow techniques applied to single images of the sample during testing to estimate the lateral (in-plane) evolution of the cracks.

Related Work
3D imaging of microstructures is possible directly through other microscopy techniques, such as scanning tunneling microscopes (STMs) and atomic force microscopes (AFMs). The former, though, requires extremely thin ma-terials (on the order of nm's), and the latter is limited to depths of ∼ 5µm and is costly. Therefore there has evolved a relatively rich set of literature exploring the reconstruction of 3D models from 2D SEM images in recent years. These include the studies of [4,5,6] for example. These studies produce disparity maps from stereo metrology, but do not specifically study the geometry of cracks.
Also, as mentioned earlier, the technique of Digital Image Correlation [3], is commonly used to study 2D surface deformation of samples undergoing fatigue testing.
Our contribution (to our knowledge) is thus to offer an approach that combines stereo imaging from the end of the experiment, with non-invasive optical flow analysis applied to mono imaging during the experiment to provide a complete 3D view of the deformation of the sample undergoing fatigue.

Dataset
All the images were provided by Prof. J. El-Awady, Associate Professor of Mechanical Engineering at John Hopkins University, co-author of [1] (and my brother) [7]. The images are all original acquisitions from his lab.  The identified angles appear consistent with visual inspection.  Next we apply epipolar analysis as presented in [11]. We begin by identifying corresponding points between the images as shown in figure (4). The variations in the shape of the cracks provide a rich environment for identifying corresponding points. In total about 50 such points were identified across the image. The computed epipoles using these points are shown in figure (5).

Epipolar Analysis
Note that experimentation showed that estimates of the epipoles was sensitive to the number of points used. Initial attempts to estimate the epipoles with less than 20 points resulted in significant variations in the estimated epipole. Once we increased corresponding points to 30 the estimate stabilized.
Using these results we can apply the Tomasi and Kanaede Factorization Method to determine the 3D structure of the scene. The resultant estimates of depths are shown in the following figures. In figure (6) we show the estimated heights / depths relative to the surface along two  Now consider all the points. In figure (7) we show the projection of the points along both the image x and y axes (i.e., the x-z plane projection and the y-z plane projection). This helps to show the range of z-displacements across the points. We see that the range of displacement is approximately ±0.2µm. We also notice a slight trend of more positive displacement (protrusion) near the top of the image (high y-value) and more indentation at the bottom. There is a very modest trend of more protrusion to the right than the left.

Result Validation
Review of these results with Prof. J. El-Awady indicate the results are very reasonable. The sample that was studied in the images was nickel, and while Prof. El-Awady does not have a direct measurement that can be used to independently to verify our estimates, there is literature where measurements have been done on other materials. [12,13,14]. These show intrusions and extrusions of the order of magnitude of 0.02 -0.3 micron in stainless steel, and as high 0.75 micron in Copper. Professor El-Awady's writes: "the conclusion is that an extrusion of 200 nm is very reasonable according to existing literature so far." Prof. El-Awady further confirmed that the result of figure (7) showing a correlation between vertical position and the z-deformation makes sense from a physical standpoint that relates to the machining of the sample that results in increased strength at the lower part of the sample.
While these comments are anecdotal, we take these are encouragement that our method produces plausible results and could be further calibrated with some direct measurements in the future.

Optical Flow and Lateral Movement
Now that we have an estimate of the surface roughness in the form of point depth estimates derived from stereo views of the final sample state, we consider the evolution of these points from the original undeformed state. This will provide insight into the lateral (x-y) movement of the points over time. Recall that while the experiment is being conducted we only have access to a single fixed front-view image of the sample.
We use the Lucas-Kanade Opticalflow method to estimate the lateral movement associated with the microcracks. We employ it using two separate sets of initial features to track. The first uses features identified using the Tomasi-Shi corner detection approach. While we will not have depth estimates for these points, they nonetheless provide a baseline for comparison of estimated lateral movement. Then we will use our corresponding points with depth estimates from section 4.2.

Lucas-Kanade Opticalflow with Tomasi-Shi
Corner Features  We experiment with the parameters of the Tomasi-Shi algorithm to find a set of 45 points that persist across all images. We find this is best controlled by the 'quality level' parameter. We set it to 0.35, along with a minimum distance of 7 and block size of 7. The trajectories of the points is shown overlayed on the images. Figure (9) shows the histogram of these lateral movements from start to finish. We see that on average the points are moving to the left, the negative x direction, by an average of 0.13 µm, and down, the positive y direction, by an average 0.64 µm. The bottom plot shows the overall inplane distance moved by the points, which averages 0.83 µm, though the distribution shows what appears to be a bimodal distribution with about half the points moving around 1.5 µm and the other half only moving about 0.5 µm.

Lucas-Kanade Opticalflow with Depth-Estimated Features
We now consider the Lucas-Kanade optical flow algorithm initialized using the set of points whose depths were estimated in section 4.2. Figure (10) shows the estimated tracks. The histogram associated with these lateral movements is shown in figure (11).
Comparing the results of figures (9) and (11) we see the shapes of the distributions and magnitudes and directions of the movements are consistent. This gives us a level of confidence in the use of these points.
It is worth noting, though, that as the reverse optical flow approaches the earliest images where the deformation is small and most of the sample is crack free, the image exhibits broad areas of homogeneous pixel values and the estimated motion appears to make relatively large jumps. This is particularly noticeable in the last (earliest) frame. This is taken to be a limitation of the approach and not a physical outcome. Now we investigate the correlation between the lateral and depth movement of the selected points. Figure (12) shows a scatter plot of the estimated lateral vs depth movement. The top left shows the z vs. total distance moved, the top right is the absolute value of z vs distance moved, the lower left is z vs x, and lower right is z vs y. From the figures we notice a strong correlation between the magnitude of the movement in the y-direction and the magnitude of the final depth of the points.

Result Validation
Discussions with Prof. J. El-Awady suggest that the lateral motion estimated from optical flow largely agrees with Figure 11. Histogram of the lateral movement of points whose depths were estimated in section 4.2. theoretical models. The cracks develop along certain slip planes within the material and these planes are purposely oriented so that all material motion happens in the y-z plane. This means that displacements in the x-direction are theoretically zero (or small) but movement in the y-direction expected. This conforms to the results presented above. The only exception to the expected results is the relatively large motions estimated for a few points in the final (ear- liest) frame due to the noted anomaly of the undeformed surface being largely of homogeneous pixel color.

Conclusion and Future work
We show how to reconstruct a 3D model of surface roughness from SEM images of a material undergoing fatigue testing. During the experiment, the camera (SEM) is fixed and produces only frontal views of the sample. At the conclusion of the experiment the sample can be detached and multiple views (at higher resolution) can be obtained. We use the views at the end of the experiment to reconstruct a 3D model of the surface roughness. Then, we use optical flow techniques to propagate the final surface back in time to obtain an estimate of the lateral movement of the points as well. Together the lateral and depth movement provide a comprehensive understanding of the surface deformation of the sample.
Experts in the field have indicated that all our estimates of both out-of-plane deformation as well as in-plane lateral deformation are of a magnitude and shape consistent with theoretical expectation. Thus it represents a promising noninvasive alternative to the current state of the art, Digital Image Correlation, which requires imprinting a speckle pattern on the sample, and is limited to just 2D estimates unless it is augmented by further costly measurements such as x-ray tomography.
A number of next steps are warranted: • The estimates should be confirmed experimentally. Tools such as atomic force microscopes can be used to obtain a measurement of the surface roughness. Samples can also be sectioned and examined to obtain depth measurements. While this is typically not done due to the cost and often limited access to such tools, the promise of the technique presented warrants performing this work.
• Once the images are obtained, the most tedious aspect was manual selection of corresponding points. An investigation of automating this process should be undertaken. Tomasi-Shi corner features appears a promising approach as a starting point for feature identification. Our experiments suggest it will select prominent features along the cracks much like I did visually. We should investigate how this can be used as part of an automated process (or at least semi-automated) to identify the corresponding points.
• Finally the surface deformation should be mapped to a strain tensor, which is the physical quantity that material scientists are ultimately seeking.