{'fraction_non_alphanumeric': 0.02302158273381295, 'fraction_numerical': 0.006096175691026127, 'mean_word_length': 5.053403621361449, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Vision Calibration Method of Robot based on Halcon

Visual calibration is an important research direction in the field of robot vision control, and is also one of the current research hotspots. In this paper, the principle of software calibration is described in detail, and a software calibration method based on Halcon optimization is studied and designed. By using the operator in the function library, the internal and external parameters of the camera are calibrated. The influence of the terminal center of the robot and the radial distortion of the camera lens is fully considered. The method is used to establish the camera. The relationship between the image coordinated system and the robot world coordinated system. Experiments show that the method has high calibration accuracy and practicability, and is suitable for industrial robot vision system calibration. Keywords—Halcon, machine vision, industrial robot, vision calibration


I. INTRODUCTION
ACHINE vision is the science and technology to simulate the function of biological vision by computer. The development of this technology is closely related to the improvement to the level of computer vision [1][2][3]. As early as 1970s, the results of the Marr comprehensive information processing, psychology and neurophysiology of the Massachusetts Institute of Technology put forward the first more complete visual computing theory from the perspective of information science, which greatly promoted the development and application of computer vision; the Marr theory of computing became a new theory in 1980s. Important theoretical framework respected by researchers in the field of computer vision. In the past thirty years, the domestic and foreign scholars have made a series of major breakthroughs in the computer vision research. A series of new theories and methods have emerged, and the progress of computer vision has laid a theoretical foundation for the industrialization of machine vision. With the continuous development of industrialization and technology, the cost of labor is increasing, and the accuracy and automation requirements of positioning assembly are higher and higher. Industrial robot positioning based on machine vision technology has many advantages, such as positioning precision, high automation and intelligent level, low labor cost and so on, so it has a wide application for the field of positioning and assembly [4][5].
The core of machine vision is graphic algorithm. Because of the huge demand of the machine vision software and hardware products, many companies have launched their own visual function library, which is convenient for the software to write directly. At present, OpenCV, Halcon and other function libraries are widely used in industrial production. Halcon is a perfect set of standard machine vision algorithm package developed by German MVtec company, which has a wide application of machine vision integrated development environment. It saves product costs, shortens the software development cycle, and facilitates the rapid development of machine vision, medical image and image analysis applications. In Europe and Japan, the industry has been recognized as the best performing machine vision software. The continuous progress of graphics algorithm provides a solid theoretical foundation for machine vision technology, and provides great impetus for the improvement to social productivity.
At present, the methods of visual calibration can be roughly divided into 3 categories: the traditional calibration method [6][7], the self calibration method and the calibration method based on the active vision [8][9].
Zhang's camera calibration method was presented in 1999 [7], and most camera calibration methods are based on this theory as a basis of improvement. Otkovic I et al used a neural network algorithm for camera calibration [10]. Kukelova et al added two constraints on the base matrix of the calibration model, which can improve the speed of the camera calibration process [11]. Paper [12] modified Zhang's calibration method based on the vanishing point principle, whereby the LM algorithm was used to simplify Zhang's calibration process, and the experiment showed that the precision of camera calibration was improved.
From above studies, we already know that the camera calibration method has been studied and improved, according to different application environments and requirements, and each method has advantages and disadvantages. This paper proposes a machine vision application for the needs of actual situation of the industrial site, which can be divided into different modules according to the function, thus forming a unified system structure. The system includes calibration module, template production module, photo trigger module, image acquisition module, image processing module and data interaction module. The software calibration is the basis of the precise measurement of machine vision.

II. MACHINE VISION SOFTWARE: HALCON
Halcon is a complete set of standard machine vision algorithm package developed by German MVtec company. It has a widely used machine vision integrated development environment. It saves product cost and shortens software development cycle. The flexible architecture of Halcon facilitates the rapid development of machine vision, medical image and image analysis applications. In Europe and Japan, the industry has been recognized as the most efficient machine vision software. Halcon originated from the academic community, it is different from the general commercial software package on the market. In fact, this is a set of image processing library, which is composed of more than 1000 independent functions and the underlying data management core. It includes all kinds of filtering, color and geometry, mathematical transformation, morphological calculation and analysis, correction, classification and identification, shape search and other basic geometric and image computing functions. Because most of these functions are not designed for specific work, Halcon's powerful calculation and analysis ability can be used to complete the work by image processing place. The application range is almost unlimited, covering medicine, remote sensing detection, monitoring, and all kinds of automatic detection in industry.
Halcon supports windows, Linux and Mac OS X operating environments, which ensures the effectiveness of the investment. The whole function library can be accessed by C, C++, C#, visual basic and Delphi. Halcon provides interfaces for a large number of image acquisition devices to ensure the independence of hardware. It provides interfaces for more than 100 industrial cameras and image capture cards, including genlcam, GigE and IIDC 1394.
Halcon 11 can realize real target recognition. The recognition method based on samples can distinguish a large number of target objects. By using this technique, the trained target can be recognized only by color or texture features, so there is no need to use one-dimensional code or two-dimensional code for target recognition.
One of the most outstanding new technologies provided by Halcon 11 is 3D surface comparison, which compares the measured surface shape of a 3D object with the expected shape. All 3D technologies provided by Halcon, such as multi view stereo vision or sheet of light, can be used for surface reconstruction; at the same time, it also supports 3D reconstruction directly through off the shelf 3D hardware scanner. In addition, aiming at the special application of surface detection, the photometric stereo vision method is improved.
In addition, Halcon now supports many 3D object processing methods, such as point cloud calculation and triangulation, shape and volume feature calculation, point cloud segmentation through section, etc. Depth from focus based depth image acquisition, fast Fourier transform (FFT) and local deformation matching of Halcon have been significantly accelerated. Halcon contains a set of interactive programming interface hdevelop, in which the Halcon program code can be directly written, modified and executed, and all variables in the calculation process can be viewed. After the design is completed, the program code such as C, C++, VB, C# , VB can be directly output and put into the program.

III. VISION CALIBRATION AND APPLICATION BASED ON HALCON
The practical significance of software calibration is to establish a mathematical model of camera imaging, which describes the process of point projection to the imaging plane in the world coordinated system. It is the ultimate goal of software calibration to accurately calculate the internal and external parameters involved in the model. The position of the target is expressed in pixel coordinates. After the calibration is completed, the coordinates in the imaging plane coordinate system can be obtained in millimeter.

A. Calibration Method of the Camera's Parameters
Camera calibration is a necessary process to accurately measure the target object. The distortion degree of each lens is different. This kind of lens distortion can be corrected by camera calibration. After camera calibration, the metric unit coordinates of the target object in the world coordinate system can be obtained. Camera calibration is actually the process of determining the internal and external parameters of the camera The Halcon function library provides a calibration method of the camera's internal and external parameters, which is simple and reliable and the calibration results is reliable. The whole calibration process includes making calibration board, parameter configuration, marking point searching and calibration calculation. By properly selecting the corresponding operators and correctly configuring parameters, we have completed the precise calibration of the camera. The calibration flow is shown in Figure 1.  The external parameters of the camera refer to the transformation parameters between the camera coordinate system and the world coordinate system. It is mainly composed of the rotation matrix R and the shift matrix T. (1) In practical application, the lens distortion correction model is derived according to the lens distortion model, such as equations (4) and (5).
In the calibration process, the high precision calibration board provided by Halcon is used, as shown in Fig. 3.

Fig.3 High precision calibration board of Halcon
Before calling the calibrate camera, a calibration data model is created by create_calib_data and the camera model is specified and internal camera parameters for all cameras are initialized by set_calib_data_camparam. The description of all calibration objects is specified by set_calib_data_ calib_object, and then the camera is used to take photos of the calibration plates placed at different positions on the detection plane.
According to the process of camera calibration, the specific process is as follows.
Step 1. A picture is collected by grab_image.
Step 2. The calibration board is found by find_calib_object.  Step 3. The mark point and outline are obtained by get_calib_data_ observ_contour.
Step 4. The center coordinates of the calibration point are extracted by get_calib_data_observ_points.
Step 5. The calibration mark is displayed by disp_circle.
The loop is executed 25 times and the camera is calibrated by calibrate_cameras.

B. Demarcated Board Making
Before calibration, the relationship between the robot coordinate system and the image coordinate system must be established through the tool fixed at the end of the robot. Compared with the traditional method of manually searching image points with fixed needle tip, the calibration board of Halcon software has the characteristics of easy extraction of calibration points, calibration accuracy and unique direction. Therefore, the calibration of camera internal and external parameters and robot vision are calibrated by Halcon calibration board. Generally, the size of the calibration plate is 1/2 to 1/3 of that of the camera. Glass calibration plate is selected for back light source, and ceramic calibration plate is selected for front light source. Printing calibration board can be selected for those with large field of vision and low calibration accuracy.
The object required for the calibration program of Halcon is the calibration board, the white bottom is made up of m × n circular marking points, the square frame of the marking point and the only triangle sign at the corner of the box. When using the calibration board to mark the timing, the area of the calibration board is required to account for about 1/3 of the whole image in principle. As shown in the common demarcation plate size, as shown in Figure 4, the marking points in the calibration board should be moderate in size, and too large will cause the adjacent marking points to be unable to separate, and the too small will cause the mark point to be unrecognized. In the selection of the calibration board, in the premise that all the marking points can be found, the distribution of the marking points should be concentrated as much as possible. The more intensive the marking points are, the more coordinates of the operations are involved, and the accuracy of the calibration is also improved accordingly.

C. Mark Point Search
In the process of calibration, it is necessary to change the position of the calibration board in the field of view of the camera constantly, change the rotation direction of the calibration board, tilt the X axis and Y axis of the calibration plate to detect the perspective deformation and lens distortion, at the same time, the surface of the calibration board is not damaged, and the accurate calibration parameters can be calculated at the end.
In the multiple regions where the segmentation is completed, the number of the hole shape area is consistent with the demarcated board description file, which is identified as the calibration plate area, and the rectangular contour in the periphery of the region is used as the calibration plate area for the summer. In order to reduce the impact of noise or other debris in the background on the extraction point, the contour screening method is used here. It is different that the closed figure less than the value is ignored and not the area.
The algorithm flow of mark point search is shown in the   All edges will be optimized to sub-pixel accuracy. The center of the mark point is stored in two arrays as output in the pixel coordinate system. The first array RCoord stores the column coordinates of all points, the second array CCoord stores the row coordinates of all points, and the values in the two arrays must correspond to the same points, and the order will be in strict accordance with the order in the demarcation board description file. Column.

D. Vision Calibration of Robot
The ultimate goal of machine vision calibration is to realize the transformation between image coordinate system and manipulator coordinate system. Camera calibration can realize the conversion between the world coordinate system and the image coordinate system, and the robot calibration can realize the conversion between the world coordinate system and the manipulator coordinate system, so as to calculate the coordinate value under the image coordinate system corresponding to the manipulator coordinate system. The calibration calculation involves complex coordinate transformation. As the most widely used coordinate system, the Descartes coordinate system is selected as the ideal position reference system in the software calibration. In any space, the establishment of coordinate system only needs to select the origin and coordinate axis. The world coordinate system is a reference coordinate system, which is used as a basis for describing the absolute and relative positions of all other coordinate systems. In some cases, it is also called the global coordinate system. In the calibration algorithm used in this paper, the origin of the world coordinate system is located in the center of the plane field in which the object is located, the Z axis is parallel to the optical axis, and the X and Y axes are parallel to the X and Y axis in the imaging plane coordinate system. In Figure 7, P(xw, yw, zw) is a point under the world coordinate system. In the camera coordinate system, it is expressed as P(xc, yc, zc), and I is the imaging plane of the camera. By the principle of the pinhole camera imaging, the point is projected on the Pi(xc, yc) point of the I plane in the I plane. The surface Pi'(xc', yc') point.
Finally, the world coordinate system is transferred to pixel coordinates.
Step 1. Transform the point P(xw,yw,zw) in the world coordinate system to P(xc,yc,zc) in the camera coordinate system, and the relationship is as follows.
This formula is an affine transformation formula. In the form, t is a translation vector that represents the position relation between the origin of two coordinate systems, and R is a 3×3 rotation matrix, which means that a three-dimensional point is rotated around the three axes of x, y and z, and the angle of rotation is expressed by the Euler angle ( ,, ).
Step 2. Convert the 3D coordinates P(xc,yc,zc) in camera coordinates to Pi(xi,yi) in the two-dimensional coordinate system of the imaging plane, which is shown as follows.
In formula (6), f is the focal length of the camera.
Step 3. The light emitted by point P in the world coordinate system refracts through the lens center, and the imaging point on the imaging plane is not Pi(xc,yc), but Pi'(xc',yc'). This is the radial distortion of the lens, and the radial distortion relation can be shown in the lower form. 2 2 In formula (7), k is a parameter to measure the degree of distortion. When k = 0, there is no distortion, k < 0 indicates a barrel distortion, and k >0 indicates a occipital distortion.
Step 4. Transform the Pi(xc, yc) in the imaging plane coordinate system to Pi(r,c) in the picture coordinate system. Assuming that Oi coordinates in the image coordinate system (r0,c0), py, and px represent the actual length of each pixel in the imaging plane yi direction and the xi direction respectively, then the formula (8) transformation relation: The unit of yi, xi, py and px in the above formula is millimeter, r, r0, c and c0 is pixel.
After the calibration results are obtained, the matching results are two-dimensional coordinates, and the returned world coordinates are also two-dimensional coordinates, and the z default is 0. This plane is the plane of the calibration plate in the calibration process, and the origin of the world coordinates is fixed as the center of the captured image by the camera.
The specific process is as follows.
Step 1. An image is captured by a camera using grab_ image.
Step 2. The image is segmented by threshold.
Step 3. Mark point is extracted by select_shape.
Step 4. The center coordinates of mark point are obtained by area_ center.
Step 5.The end tools of the moving manipulator are aligned with the corresponding mark points to obtain the corresponding manipulator coordinates.
Step 6. The affine transformation matrix of image coordinates and mechanical coordinates is created by vector_to_hom_mat2d.
Step 7. The affine matrix parameters are obtained by hom_mat2d_to_affine_par.

A. Experimental Demarcation
In this paper, the experiment is carried out on the rectangular coordinate experimental platform. The camera adopts the industrial camera of Hikvision model MV-CE013-50GM. A ring light source is fixed under the camera. The HALCON calibration board LHC025-1-7 × 7 is used, and the field of view is 50 × 40. The accuracy of the panoramic camera used in the experiment is 1024×768.
In the calibration process, HALCON software is used to extract corner points in the image. The calibration board is shown in Figure 8. The checkerboard [12] of 7×7 is selected. The size of each chessboard is 30 mm×30 mm, and the corner point extracted from each picture is 6 × 6 = 36. Select 10 pictures in different positions and angles.
According to the algorithm flow shown in the figure, 10 images of the calibration board are collected for camera calibration, and the camera calibration parameters are obtained, as shown in Table 1 and Table2.  In HALCON image processing software, there are many operators of corner coordinates extraction. For points_foerstner, points_harris, and points_lepetit, several operators can extract cross points, but it can not exclude some corner coordinates of the edge of the calibration board, and the coordinate error is larger. Figure 8 shows the results of extracting corners from several operators. In this paper, the saddle_points_sub_pix operator is used to extract the corner points. It has the following characteristics: (1) the accurate extraction of diagonal points is realized through the mathematical analysis of sub-pixel precision (partial derivative level); (2) the relative uniform image gray value region is excluded by using the threshold of the curvature of the gray value function. The flow chart of software calibration program for main thread and background thread is shown in Figure 9. After the demarcation begins, the calibration board is first made, the demarcated board description file is read in, and the "collection" button is pressed to judge whether it is first shot, if it is initialized the camera internal reference, otherwise the picture processing mark position is Capture is set to be true. If enough photographs are taken, the calibration button can be clicked, when the background thread will store multiple sets of mark coordinates, start the calibration calculation, get the inside and outside parameters of the camera and save it.   Figure 10 shows the comparison between the calibration error of the literature [13]~ [15] and the calibration error of this paper. It can be seen that the calibration accuracy is high.

B. Precision Measurement
In the precision test, the template of the target object is input first. Through accurate measurement, the target object is placed in a fixed position in the camera's field of view. After the camera takes pictures, the matching result is recorded. The results of the experiment are shown in Table 3, and the error analysis is shown as in Figure 8. According to the test results, the measurement error of the visual software is less than 0.35mm in the direction of X, 0.25mm in the direction of Y and 0.3mm in the direction of Z. The matching accruancy of calibration meets the requirements of the system.
Using the powerful visual algorithm library of Halcon, the collected images are processed and calibrated to obtain the average error of the calibration results. The relationship between the image coordinate system and the mechanical coordinate system is determined by establishing the homogeneous transformation matrix between the image coordinate system and the mechanical coordinate system. The results show that the calibration method can calculate the bevel angle accurately and meet the requirements of industrial robot.
The multi-sensor fusion technology is introduced into the robot vision research, and the position information obtained has the characteristics of redundancy, complementarity and realtime. Although capacitive, mechanical and optical sensors are all used in practice, it is difficult to complete effective visual calibration with conventional sensors in complex environment because the detection range of non visual sensors is not as complete as that of visual system. Therefore, developing new image sensors, positioning sensors or constructing sensor array according to certain fusion strategies to make up for the defects of single sensor, and proposing new fusion methods to improve the results of visual calibration will be important research directions in the future.
Under the condition of supervision, the biggest disadvantage of the learning network is that it has to relearn after the position changes, which is difficult to apply to robots with incomplete environment information or frequently changed environment. Therefore, it is necessary to develop an adaptive neural network with higher intelligence, self-learning ability under unsupervised condition, and using fuzzy rules to guide learning and restart when the environment changes.
In the robot vision calibration system, using a single intelligent method can not achieve satisfactory results. An ideal vision system should include three levels of execution, coordination and management, and each level should integrate conventional and intelligent methods to achieve good results. Neural network and fuzzy reasoning are two important tools in robot vision research. However, it is very difficult for robots to obtain a complete neural network sample set, and it is obviously not advisable to take every point in the event space as the learning sample of the network. Fuzzy logic reasoning focuses on the selection of fuzzy rules, but some rules are difficult to describe formally, or they must be described by a large number of rules to increase the amount of calculation, which deviates from the original intention of fuzzy logic. Therefore, it will be a hot research field to combine fuzzy logic with neural network to improve the adaptability and robustness of robot.

V. CONCLUSION
From the perspective of visual positioning, this paper studies and designs an optimized method of software calibration. By using the operator in the function library, the internal and external parameters of the camera are calibrated. The technology of computer vision positioning based on template matching is used. First, image preprocessing and corner point coordinated extraction are used by HALCON. Compared with other extraction processes, the accuracy and efficiency is improved. By calculating the accurate position of the object, the robot is guided to grasp the object of the work scene intelligently, and the efficiency and quality of modern industrial production can be improved by introducing the computer vision positioning technology. The testing accuracy, matching speed and operation stability of the software meet the requirements of the system. It is suitable for the robot vision system that needs to be captured and assembled in industry. Improving the adaptability of visual software and improving the speed of algorithm matching are the main concerns in the next stage.