{'fraction_non_alphanumeric': 0.024196101487947177, 'fraction_numerical': 0.008909592510515128, 'mean_word_length': 4.832761804273279, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Vision-Guided Active Tactile Perception for Crack Detection and Reconstruction

Crack detection is of great significance for monitoring the integrity and well-being of the infrastructure such as bridges and underground pipelines, which are harsh environments for people to access. In recent years, computer vision techniques have been applied in detecting cracks in concrete structures. However, they suffer from variances in light conditions and shadows, lacking robustness and resulting in many false positives. To address the uncertainty in vision, human inspectors actively touch the surface of the structures, guided by vision, which has not been explored in autonomous crack detection. In this paper, we propose a novel approach to detect and reconstruct cracks in concrete structures using vision-guided active tactile perception. Given an RGB-D image of a structure, the rough profile of the crack in the structure surface will first be segmented with a fine-tuned Deep Convolutional Neural Networks, and a set of contact points are generated to guide the collection of tactile images by a camera-based optical tactile sensor. When contacts are made, a pixel-wise mask of the crack can be obtained from the tactile images and therefore the profile of the crack can be refined by aligning the RGB-D image and the tactile images. Extensive experiment results have shown that the proposed method improves the effectiveness and robustness of crack detection and reconstruction significantly, compared to crack detection with vision only, and has the potential to enable robots to help humans with the inspection and repair of the concrete infrastructure.


I. INTRODUCTION
Cracks in the infrastructures surface are important indicators for assessing the condition of buildings and need to be repaired timely for preventing the expansion of potential risks. However, manual crack detection and maintenance are not only time-consuming and expensive, but also pose health risks to human workers in harsh and complex environments such as dams and underground pipelines. Hence, development of an effective and robust crack detection system will be significant for the substitution of inspection workers and the development of smart cities [1].
During the past two decades, the problem of crack detection attracts wide attention from researchers and has been widely explored and developed [2]. However, most of the previous works address this task based on only one single modality such as acoustic data [3] [4], laser-scanned image [5], optical fiber signal [6], and RGB images [7]. In this case, those methods using the high-precision modality such as acoustic data cannot perceive objects quickly due to the limitation of the small receptive field. Moreover, they could not correct the detection results with additional information All  if a crack is not detected or a noisy pattern is falsely detected as a crack.
When humans intend to perceive objects, they gather information and draw a solid conclusion with their eyes, ears and hands together. In crack detection tasks, skilled inspectors usually first look at the surface to find areas with similar color or shape characteristics to cracks, and then use their fingers or specific tools such as a hammer or ultrasonic device to further inspect those areas instead of traversing all regions. In this way, they can fully utilize the properties of different modalities including the efficiency of visual sense with a large receptive field and the accurateness of tactile sense or acoustic sense.
Inspired by those observations, we propose a novel visionguided active tactile perception approach for crack detection and reconstruction in this paper. This method integrates visual and tactile senses so that they aid each other and improve the perception on cracks. On the one hand, visual sense with a large receptive field can achieve a quick search for finding the candidate regions containing cracks, which can reduce the area of tactile perception and improve its effectiveness. On the other hand, tactile sense can enhance the robustness of crack detection thanks to the characteristics of being less susceptible to light and noise, and can reconstruct the crack shape precisely with high-resolution tactile images obtained by a camera based optical tactile sensor.
To evaluate our proposed method, we collected a visual dataset and a tactile dataset with 10 mock-up structures that were manufactured using 3D printing. Half of those objects only contain real cracks, and the rest contain both real cracks and fake cracks that are paintings on the surface. The extensive experiment results demonstrate that our approach is capable of distinguishing real cracks from fake cracks, and reconstructing the crack shape quickly and accurately. Compared to the visual method, our approach achieves a significant reduction of mean distance error from 0.82mm to 0.24mm. Furthermore, the proposed method is more than 10 times faster than passive tactile perception in terms of tactile data collection time.
Our contributions can be summarised as follows: • A novel vision-guided active tactile perception framework is proposed, for the first time, for crack detection and reconstruction; • A touch point generation method is designed for the vision-guided active tactile perception; • A tactile crack perception method is developed that can suppress the false positives of visual detection and reconstruct the crack with a millimeter-level resolution.

A. Vision-Based Crack Detection
Recently the study of image-based crack detection has drawn increasing attention from researchers in the fields of both Computer Vision and Civil Engineering. Early visionbased cracks detection methods mainly rely on conventional image processing methods such as edge detection [8] [9] and thresholding [10] [11]. However, those methods are susceptible to light changes and scanning noises due to the subjective parameter selections for pre-processing and edge detection. Moreover, there are also works using conventional machine learning approaches such as Support Vector Machine [12] [13], Ada-Boost [14] and Naïve Bayes classifier [15]. Nonetheless, the features used in these traditional machine learning methods such as LBP, wavelet and histogram can only represent one or two layers of abstraction, which are unable to capture features of cracks in surfaces with complex backgrounds.
In the recent years, works using deep neural networks for crack detection have emerged [16], [17]. Based on the fineness of detection results, those deep learning-based methods can be summarised as two types of approaches: box-level detection and pixel-wise detection. Typical boxlevel crack detection approaches firstly generate a set of rectangular frames to locate the cracks using patch scanning [16] or region-based CNN networks [7] and then filter out false positives with a Naïve Bayes classifier [16] or a Bayesian integration algorithm [7]. However, the rectangle shape results are not precise and are limited with regard to complex crack distribution and irregular infrastructure surface. The more recent development in deep learning-based defects perception is pixel-wise crack detection. In [18], a novel encoder-decoder network is proposed, which uses hierarchical convolutional feature learning to separate the cracks from background. In [19], a deep generative adversarial network is proposed for pavement crack detection, which is trained end-to-end with a crack-patch-only supervised method to overcome the local minimum problem.

B. Active Tactile Perception
Tactile sensing plays an important role in robot perception and has been applied to a number of different tasks such as object recognition [20], material property analysis [21], and shape exploration [22]. Most early studies related to tactile perception are based on passive data acquisition processes and are considered simply as a forward process [21], [23]. Nonetheless, humans usually plan their actions with a goal and actively refine their sensations (especially the tactile sensation) rather than passively gather information. Inspired by this, there has been growing interest in active tactile perception [24], [25]. In [26], an active tactile perception method using pre-specified control laws was proposed for clothing material recognition.
However, there has been no works on crack detection with active tactile perception yet. In [6], tactile sensing was explored for crack detection and characterization, but in this work tactile data was collected passively and the position of the crack cannot be localized accurately. Our proposed vision-guided active tactile perception is the first work that incorporates visual information into active tactile perception for crack detection and reconstruction. It can actively perceive the crack region with the guidance of vision, and reconstruct crack shape precisely with a highresolution camera-based tactile sensor.

III. METHODOLOGY
In this work, we propose a vision-guided active tactile perception framework that can detect and reconstruct cracks efficiently and precisely, with an overview of the framework illustrated in Fig. 1. Given a single RGB-D image of cracks in a structure, our method first uses the color image as input to a deep convolutional network to predict a mask of the cracks. Then the predicted mask and the depth image are used to generate a set of contact points, which directs the collection of the tactile images with a camerabased tactile sensor. Another deep convolutional network is applied to predict the masks of cracks in the collected tactile images that are used to refine the visual detection result and reconstruct an accurate cracks profile.

A. Visual Guidance for Touch
A deep semantic segmentation network is first utilized to predict pixel-wise masks of cracks. To guide the touch sensing, the contact points with cracks skeletons are then generated. Details of these two steps are given below.
Visual Crack Segmentation. The visual crack segmentation is treated as a semantic segmentation problem that predicts each pixel of the input image into one of two semantic classes: (a) background (b) cracks. To this end, we use the Deeplabv3+ model [27] to segment the cracks in visual images that is a state-of-art deep learning model for semantic image segmentation. A ResNet-101 backbone [28] is used. Since the number of pixels of the background is much larger than that of the cracks, the network may easily converge to the status that treats all the pixels as background. To address this issue, we use the original images as input instead of resizing it to a smaller size as done in the previous works [17] and use a weighted cross-entropy loss with crack pixels weighted 10x more than background pixels. Moreover, we set the output stride value as 8 as smaller values give finer details in the output mask. In the training process, we start with a model pre-trained for semantic segmentation on the COCO dataset [29] and use the following hyperparameters: SGD Optimizer with constant learning late of 1e-6, momentum 0.9 and weight decay 5e-4.
Contact Points Generation. Given the predicted pixelwise crack mask in the color image, we can extract the skeleton of each crack mask with pattern thinning method [30]. We define two types of keypoints (i.e., end points and branch points) and minimal edges that represent the topology of the crack pattern: Another deep convolutional network is used to segment the crack in the collected tactile images. Given the detected tactile crack mask, the crack shape are reconstructed based on the geometrical model of the GelSight sensor and the coordinate transforming relation between the tactile sensor coordinate and the world coordinate.
• Minimal edge E ij : if there is a continuous path between two keypoints p i and p j and all points on the path are neither end points nor branch points. For every minimal edge E ij which consists of a number of ordered points, the keypoint p i is initially selected as the current contact point p current . Then we iteratively choose the next contact point p k using the following formula: where D(p current , p k ) is the distance between two points in world frame. The hyper-parameter d is the threshold of the distance between two points that is related to the coverage and speed of tactile perception. A smaller d will increase the coverage while reducing the perception speed. In our case, d is empirically set to four fifths of the tactile sensor's view length. As shown in Fig. 1, the end-point pixels and the generated contact points are tagged with red dots and green dots, respectively. For each contact points p i , the yaw angle of the end-effector is parallel to the vector < p i , p n >, where p n is the nearest contact point to p i , so that the end-effector can contact the surface perpendicularly.

B. Active Tactile Crack Perception
Tactile Crack Detection. To address the problem of false positives in the visual crack detection due to light changes and shadows, we apply tactile information to refine the vision-based detection results and reconstruct cracks shape in 3D space. Firstly, we control the robotic arm equipped with a camera based Gelsight tactile sensor [26] to collect tactile images autonomously at the generated contact points in Section III-A.
The GelSight sensor is a camera-based optical tactile sensor that can capture fine details of the object surface. As shown in Fig. 2, a webcam 1 is placed under an elastomer and captures the deformations of the elastomer when it interacts with the object. The sensor has a flat surface and a view range of 14mm × 10.5mm and can capture tactile images at a frequency of 30 Hz [31].  The collected tactile images are fed to another deep convolutional network (Deeplabv3+ [27] with ResNet-101 [28]). Since the number of background pixels is similar to that of the background pixels in tactile images, we use the vanilla cross-entropy loss instead of the weighted crossentropy loss in Section III-A. Using the predicted masks in tactile images, we can double check the visual segmentation results. Any minimal edge will be set as false positives and deleted from visual segmentation results, if there are more than two tactile images whose predicted crack area is smaller than a predefined threshold. The threshold is empirically set as one fiftieth of the total number of pixels in the tactile image.
Tactile Crack Reconstruction. In the task of crack perception, it is crucial to obtain the shape and size of the crack when assessing its potential risk to the building and infrastructure. However, limited by the accuracy of the depth camera, current vision-based reconstruction methods are not able to reconstruct small cracks precisely. To this end, we use tactile images obtained from the GelSight sensor to reconstruct the cracks, whose spatial resolution is about 20 to 30 microns.
First, we predict the location of cracks on the surface of the GelSight sensor, given the detected boundaries of pixel-wise masks in the tactile images. To simplify the problem, we model the webcam using a pinhole camera model, and treat the surface of GelSight sensor as a flat plane that is perpendicular to the webcam's z axis. Hence, the transformation between a contact point P = [X c , Y c , Z c ] T in the tactile sensor coordinates (the tactile sensor take the centre of the webcam as the origin) to the pixel P = [u, v] T in the tactile image coordinates can be calculated: where Z c is the distance from the optical center to the elastomer surface of the tactile sensor. K is the matrix of the intrinsic parameters of the webcam and can be represented as: where f is the focal length of the webcam, d x and d y denote the pixel size, and (u 0 , v 0 ) is the center point of the tactile image.
After obtaining the position P in the tactile sensor coordinates, we can calculate its position P W in the world frame: where T E C and T W E are the transformation matrix from the tactile sensor coordinates to the end-effector coordinate system, and from the end-effector coordinate system to the world coordinate system, respectively.

IV. EXPERIMENT SETUP
In this section, we introduce the robot setup used for data collection and experiments. An overview of our robot setup is shown in Fig. 3.

A. 3D Printed Structures with Cracks
In this work, we follow part of the data acquisition protocol in [6]. A set of 10 structures with cracks of different widths (holes in the structures) were manufactured with PLA plastic using an i3 Mega 3D printer from ANYCUBIC. To test the robustness of the proposed method, we also paint several fake cracks on the surfaces of the structures. The samples are shown in Fig. 4.

B. Visual Data Collection
We put those 3D-printed structures on a table and took 10 images of each structure with real cracks that are used for training the visual model. Then we take 3 images for each object separately for the test. All images were taken with a RealSense D435i camera.

C. Tactile Data Collection
The setup of tactile data collection is composed of two parts: a 6-DOF UR5 collaborative robot arm from Universal Robots and a GelSight sensor mounted on a 3D-printed end-effector. To collect the tactile data autonomously and repeatedly, we build a data collection software using Robot Operating System (ROS). The software can control the robot arm to move across the surface of a structure following the pre-defined initial position, steps, and step length in x-axis and y-axis. As for each position, we rotate the sensor about the axis perpendicular to the surface with different angles so that the dataset can be more generalized. As for each contact, the robot arm will stop moving and the GelSight records one tactile image when the pressure reaches a threshold. In this way, we can capture tactile data of good quality and also avoid the unnecessary protective stops of the robot arm. In total, 544 valid tactile images were collected and split into training and test sets (370 and 174 for each, respectively), with some samples and their annotations shown in Fig. 5.

A. Experiments on Network Architectures
To evaluate the accuracy of the tactile crack segmentation, we use both Intersection over Union (IoU) and True Positive rate (TP). The results of different models and input image sizes are listed in Table I. Two different backbones was used for Deeplabv3+: ResNet-101 and DRN-54 (Dilated Residual Network). Example segmentation results of ResNet-101 and DRN-54 with an input size of 256×256 are shown in Fig.  6. The results indicate that compared to Resnet-101, DRN introduces more noise over training and perform worse for the tactile crack segmentation due to the fact that it has a larger receptive field.
We also experimented with different input image sizes. The results show that a smaller input image size leads to better performance, which is surprising as images of a higher resolution may contain more details. One possible reason is that a certain amount of noise is generated when resizing the width of the tactile image from 480 to 512.

B. Crack Detection Results
Similar to the evaluation method used in network architectures experiments, we evaluate our proposed methods based on standard evaluation metrics of pixel accuracy (pixAcc) and IoU. The segmentation results of using only visual information and using both visual and tactile information are summarized in Table II. The results show that both the pixel accuracy and IoU of visual semantic segmentation have a significant drop due to the existence of fake painting cracks from 0.899 to 0.866 and from 0.504 to 0.376, respectively. It is also shown that after using tactile information to find the fake paintings and refine the visual detection results, the performance of cracks detection has been improved effectively. Due to the fact that cracks on the RGB images only take up fewer pixels compared to background, the pixel accuracy does not change dramatically as IoU.

C. Reconstruction Results
As for the crack reconstruction task, we use the mean, max and standard deviation (SD) of the shortest distance between the actual crack shape and the reconstructed crack location to evaluate the accuracy of our proposed method. There are four methods used for comparison. The vision method uses point clouds recovered through visual detection and depth information to represent cracks. In order to reduce the impact of depth information accuracy on reconstruction, the aligned vision method projects the point cloud to the table surface. The passive tactile method collects the tactile images through traversing the whole 3D printed structure surface.
As can be seen in Table III, our approach shows a significant improvement of mean distance error from 0.55mm to 0.24mm compared to the aligned vision method. Two example reconstructed crack profiles are shown in Fig. 7, which shows that reconstructed crack profiles with tactile data is much closer to the ground truth compared to the visionbased method. On the other side, compared to passive tactile perception, our proposed vision-guided tactile perception is more than 10 times faster than passive tactile perception in terms of the tactile data collection time without affecting the accuracy much.

VI. CONCLUSION AND FUTURE WORK
In this paper we introduce a novel vision-guided active tactile perception for crack detection and reconstruction. The cooperation between those two modality addresses the false positives in visual detection results. The experiments show that our proposed method can improve the effectiveness and robustness of crack detection and reconstruction significantly, compared to when only vision is used. It has the potential to enable robots to inspect and repair of the concrete infrastructure. Future works to improve our method can also be considered, such as cracks detection in curved surfaces, the use of weakly supervised learning methods to segment cracks and addressing the localization uncertainty of the robot arm in active tactile exploration.