{'fraction_non_alphanumeric': 0.034408602150537634, 'fraction_numerical': 0.009901433691756273, 'mean_word_length': 4.645169448659585, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Auto-calibration Method Using Stop Signs for Urban Autonomous Driving Applications

For use of cameras on an intelligent vehicle, driving over a major bump could challenge the calibration. It is then of interest to do dynamic calibration. What structures can be used for calibration? How about using traffic signs that you recognize? In this paper an approach is presented for dynamic camera calibration based on recognition of stop signs. The detection is performed based on convolutional neural networks (CNNs). A recognized sign is modeled as a polygon and matched to a model. Parameters are tracked over time. Experimental results show clear convergence and improved performance for the calibration.


I. INTRODUCTION
Recently, the expectations placed on autonomous driving have risen sharply, and many big companies and start-ups have launched their own production lines with different architectures. The needs of safe and comfortable driving give the advanced driver assistance systems (ADAS) an essential role in the development. Among all kinds of sensors supporting ADAS, the camera stands out for its low-cost and capability of capturing the scenarios with rich contexts. A camera system is frequently used in distance perception, speed inference, and object detection [1]. For these applications, the camera calibration parameters are often essential to reconstruct parts of a scene and can influence the accuracy of the measurements.
One can calibrate a camera once before the beginning of a task that requires minimal camera motion, as the intrinsic parameters are reasonably assumed to be unchanged. However, when the camera is mounted on a vehicle, the camera intrinsic parameters turns out to vary due to the mechanical vibrations caused by the rough street surfaces, or temperature fluctuations caused by the weather [2], [3], [4]. It is inconvenient to repeatedly calibrate a camera for a long-term application, and not practical throwing the technical task to an innocent customer, with the goal of popularizing autonomous vehicles. The long-term operations and the maintenance-free advantages of a camera system can only be guaranteed by auto-calibration methods, which can update the camera parameters continuously.
The commonly used camera calibration methods (e.g. [5]) require some special reference patterns, a chessboard, for example, whose physical size is known beforehand. However, these special patterns are not available when the vehicle is on the road. In this case, the auto-calibration system is necessary to extract the reference information from its environment. There are a few auto/self-calibration methods proposed. Ribeiro [6], Hai [7], and Lu [8] extracts feature points from the road lines for calibration purpose, yet it is doubtable to assume the same width of the road lines in different regions. Also, because of frequent friction (with wheels), imperfections exist along the contours of road lines. These artifacts and variations can lead to undesirable calibration errors. In [9], two solutions are presented. One designs a structured environment (a grid of known size painted on the road) that the cameras detect as the vehicle drives near, the other puts several markers on the vehicle's hood, which may distract the drivers. Also, the hood is curved in most cases and is assumed to be within the FOV of the camera. Both are not suitable for real driving scenarios.
Stop signs, unlike a designed pattern, are ubiquitous ( Fig.  1). The physical size of stop signs are standardized by the U.S. Department of Transportation [10].
Compared with other reference information existing in the urban environments, there are three advantages of using a stop sign: • There are a number of adequate feature points that can be detected with sub-pixel accuracy (at least eight from the inner octagon corners of the stop sign). • Compared with other reference objects (e.g., road lines), the relative geometric properties of a stop sign remain unchanged anywhere anytime, because they are made of metals which are resistant to external forces. • A vehicle slows down when approaching a stop sign, reducing the image blurs and the rolling shutter effects.
Therefore, we present an approach to auto-calibrate with stop signs. The rest of the paper is organized as follows: Section II gives a brief summary of the related work; Section III presents the details of our approach; Section IV shows the experiment results on the videos collected from the UCSD campus using four vehicular cameras.

II. RELATED WORK
Camera Calibration: Camera parameters can be estimated from enough pairs of 3D object points and corresponding 2D image points [11]. In [12], Zhang proposed to calibrate a camera with a planar object, opening the door to the famous chessboard calibration method [13].
Auto-calibration: Several researchers have also contributed to the auto-calibration using different reference objects. In [6][7] [8], vanishing points or road corners were extracted from the dotted road lines for online calibration. In [14], predefined feature points from other automobiles were used to build the correspondence model, applied to obtain traffic camera parameters. In [15], surveillance camera calibration was based on three vanishing points defining the stream of vehicles. In [16], the same work was done through representing a walking human as a vertical line segment. Besides, [9] placed markers on the vehicle's hood, from which the relative positions and orientations of the on-board cameras can be estimated quickly.
Traffic Sign Detection: Object detection has been a well-studied area with the help of convolutional neural networks (CNNs). Proposal-free architectures (e.g. YOLO [17], YOLOv3 [18]) can easily run in real-time frequency of 45Hz, while proposal-based ones (e.g. Faster R-CNN [19], FPN [20]), though requiring more resources and typically taking longer to run, generally outperform the formers in terms of accuracy, especially with small objects and complicated scenes. As to tackle a related task, the instance segmentation method in [21], which labels the object pixels, can also be applied to find the stop sign bounding boxes. Besides, optimized architectures (e.g. [22], [23]) tailored for the traffic sign detection have been proposed as well.
Shape Matching: Shape matching checks the resemblance of two shapes using some similarity measure, or a metric [24]. Particularly in our case, the shape enclosed by the detected stop sign edges has to be an octagon. Arkin et al. [25] represented the polygons with the turning functions and measured the L p distances. Huttenlocher and Kedem [26] moved further by considering (in addition to rotation, translation, and scale invariance) the affine invariance, with a Hausdorff distance on top of the normalized affine arc length representation of the polygon vertices. Later, several algorithms (e.g. [27], [28]) were introduced to speed up the calculation of the Hausdorff distance.

III. METHODOLOGY
The automatic system can be divided into seven modules (see Fig. 2).
The first five parts generate the 2D-3D correspondence pairs of corner points from an urban video or image sequence, followed by the last two steps calculating the intrinsic parameters.

A. Stop Sign Detection
A Mask R-CNN model with a ResNet-50 and Feature Pyramid Network backbone [29] were used to generate 2D bounding boxes on the image frames (see Fig. 2(a)). The network was pre-trained on COCO2017 [30].

B. Contour Points Detection through Color Segmentation
This module detects the contour points of the red inner octagon from a stop sign ( Fig. 2(b)). An image pre-processing of color space conversion (from RGB to HSV) proves to be beneficial for the red color extraction with a threshold. Afterwards, the Canny/Devernay algorithm is used to attain the contour points of the octagon with sub-pixel accuracy: a. The Canny Algorithm Canny [31] applies the first derivative of a Gaussian filter over an image to get the image gradient, of which the edge points are the local maxima along the normal direction n. The direction n orthogonal to an edge can be approximated with the image gradient: where G and I are the Gaussian filter and the image, respectively, and stands for the convolution operator. b. The Devernay Sub-Pixel Correction Edge points are attained with pixel precision through the Canny Algorithm, yet sub-pixel accuracy is desired in our system. This is where the Devernay Correction [32] comes as an improvement of the former. Suppose B is a pixel point obtained with the Canny Algorithm (a local gradient maximum in the corresponding normal direction n). The gradients of two neighbouring points of B along n can be approximated through linear interpolation. Then, the sub-pixel position of the edge point is refined to the maximum of a 1D quadratic interpolation (along n) of all these three points. c. Contour Points Chaining The above independent points can be grouped to form several chains [33], among which the one contains the most points is selected as the contour of the inner octagon, assuming a dense point detection (otherwise, the contour may be broken up into several pieces).

C. Edge Line Fitting
Given the set of 2D points X ∈ R N ×2 on the stop sign contour from above, eight lines are to be fitted, with each estimating an edge of the octagon.
The edges are estimated one at a time, by finding the line with the most "support points". A contour point "supports" a line when the point-to-line distance is within a closeness threshold (in this work, the threshold is fixed to 0.5 px to assure a sub-pixel accurate estimation). This line selection is done by running a RANSAC algorithm over pairs of contour points (Algorithm 1).
Randomly sample two contour points; Fit a line through them; Find and count the support points of the line above; Record the support points whenever the count is the highest so far; end Find the least square line of the recorded support points. Determine the two end points of the line segment from the extrema of the recorded support points.
With p ∈ [0, 1) probability that the final least square line is the most supported line, the number of iterations K can be calculated as [11]: where w = 1 (8−i) 2 , i ∈ {0, ..., 7} is the chance of selecting a point from the support set, assuming that there are few outliers in X off the contour.
Once an edge has been fitted, all the support points should be removed from the original set, so that the same Algorithm 1 can be repeated to find a different edge.

D. Line Refinement with Perpendicular Search
Primitive points from section III-B can stray away from the contour due to misleading color segmentation in the HSV space caused by poor illumination. Line refinement is therefore necessary to adjust the shifted lines back to the edges through a local search.
The magnitude of an image gradient reveals the position of the octagon contour, where the color changes most abruptly in the neighborhood. Starting from an estimated line of section III-C, we search for the local maximal gradient along the perpendicular direction of the line, and rotate the line to align with it (Algorithm 2). In the case of under-/over-exposure, the gradient peak of an edge may disappear. Thus, a refinement boundary is introduced to cope with this. With the assumption that the refinement should be subtle, a line is only updated when more than half of the S refined points are within the boundary. The refinement boundary is parameterized with a percentage B ∈ [0, 1], meaning that a refined point is at most 100B percent of the white border width away from the original position along the perpendicular direction. To estimate the white border width of a stop sign, we multiply the length of an estimated line segment from Algorithm 1 with the width-to-length ratio of an inner polygon edge documented in the traffic sign regulation.

E. Estimation of Intersection Points
There are, at most, 8 2 = 28 intersection points from the eight estimated lines, while only eight of them are valid octagon corners. A simple yet empirically effective approach would be to select the intersection points closest to the end points of the line segments. After this, any estimation with a corner count not equal to eight is dropped.
Sorting the corner points in, for example, a clock-wise order helps obtain the 2D-3D correspondence. Further, the first corner can always be set as the one to the upper-left of the character "S", for stop signs generally stand upright.
It is never too cautious to check the shape. Hausdorff distance method in [26] is adopted to guarantee that the final eight-point set forms an octagon under an affine transformation, which is, though two degrees-of-freedom less than the actual projective transformation, the best we can do without getting into the chicken-or-egg dilemma.

F. Calibration with Planar Objects
The pinhole camera model [34] is used. A 3D point (X, Y, Z) in the world coordinates is projected into an image as a pixel (u, v). The following equation relates (u, v) to (X, Y, Z): where s is an unknown scalar and A ∈ R 3×3 is the camera intrinsic matrix. r 1 , r 2 , r 3 are the columns of the rotation matrix, and t is the translation vector. They are often called the camera extrinsic parameters, for their function of transforming a world coordinate point to the camera coordinate system.
If the 3D coordinate system is fixed on a planar object like a chessboard or a stop sign, the Z component in Equ. 3 is always zero, which leads to a simplified equation: We use M = (X, Y, 1) T and m = (u, v, 1) T to represent the position of a point on the planar object and its position in the image, respectively. Their relationship is defined by a homography matrix H: Based on the knowledge that r 1 , r 2 are orthonormal, we have: These are the two constraints needed for computing the intrinsic matrix A. The methods of solving the camera calibration problem and obtaining the closed-form solution are described in [12]. We refer interested readers to that paper for more information.
In our system, assuming that the principal point is fixed at the center of the image plane, only the focal lengths f x and f y are computed and iteratively updated as shown in the next section. For chessboard calibration, we also enforce this assumption for the purpose of comparison.
The closed-form solution can be refined through maximum likelihood inference: The original calibration is used as the initial guess for the nonlinear optimization problem, which can be solved with the Levenberg-Marquardt Algorithm as implemented in [37]. Currently, this part is not included in our system.

G. Temporal Updates of Calibration with a Kalman Filter
The objective is to integrate intrinsic parameters over time. It is reasonable to assume that the errors are due to additive Gaussian noises, thus we use a Kalman filter.
The prediction step is: where x t , x t+1 are two (vectors of) intrinsic parameters estimated at different times. As there is no systematic manipulation over the intrinsic parameters, we can leave out the term Gu t . If we believe that the intrinsic parameters are constant within a short period of estimation (which is a fundamental assumption of our research), F will just be the identity matrix. The update step is: which explains "given the current estimated intrinsic parameters x t , what might be the true estimation of these parameters p t right now". For this,H is again an identity matrix. The process noise and the measurement noise δ are Gaussian random vectors with zero means. Their covariances are denoted by (diagonal) matrices Q = E( T ), R = E(δδ T ), respectively. Q specifies how much we think the intrinsic parameters change over time, which should be low for a short estimating period. R determines how much noise there is in one estimation. Intuitively, this measurement noise ought to decrease from a high value as the process goes, because the more images are used, a more accurate calibration is expected. Our experiment data was collected by one of our experimental autonomous vehicles. The vehicle is equipped with four cameras. The cameras are set up as two on the front and one on each side (Fig. 3). Videos of the cameras, data of LiDARs, and the vehicle positions were recorded for experiments, by driving along multiple routes around the UC San Diego campus [38]. The camera video was streamed at approximately 12Hz. The camera data are used for urban scenario experiments 1 .

B. Datasets
The experiments were done using two datasets collected by the Autonomous Vehicle Lab (AVL). The video frames were stored in two different formats: a. StopSignV1 -PNG Images Converted from Compressed JPG Images Camera 1 and 5 detected 227 and 515 stop sign candidates from the video frames, respectively, and camera 6 detected 100 and 130 stop sign candidates at different times. The camera 2 images are too few to get a good calibration result, because stop signs are often found on the opposite side (right-hand side of the vehicle based on U.S. traffic rules). The ground-truth parameters (GT) were obtained from chessboard calibration before the video/image collection. b. StopSignV2 -Uncompressed PNG Images Only camera 5 and 6 were used, each detected 1507 and 1330 stop sign candidates, respectively. The groundtruth parameters (GT) were obtained from chessboard calibration beforehand.

C. Results
For scale-free comparison, the relative error f of each parameter f ∈ {f x , f y } is computed: where f GT ∈ {f GT x , f GT y } is the ground-truth intrinsic parameter.
a. StopSignV1 -Compressed PNG Images For all of the four experiments, the refinement boundary percentages were the same: B = 0.4. Right before the calibration module, the system selected 98, 93, 66, and 93 good stop signs for experiments Camera1, Camera5, Camera6-Time1, and Camera6-Time2, respectively. We present both the tracked outputs of the complete system (red curves) and the pure calibration results (blue curves) as the number of image frames increases (Fig. 4 below and Fig. 9 in Appendix). To show that the system can attain reasonable calibration parameters without the Kalman filter, for each camera dataset with N stop signs, we repeatedly remove one (different) stop sign at a time and calibrate using the remaining N − 1 stop signs. The mean relative error (number above each column) and the corresponding standard deviation (red error bars) of these N calibrations are calculated (Fig. 5).

D. Discussions
A 5% relative error is achieved for StopSignV2-Camera5 (Fig. 7), which has over 400 stop signs for calibration. If we consider the 1-2% variation of the chessboard calibration (which is verified by experiments), the above error can be as low as 3%. This is just 1% higher than that of the chessboard method.
Comparing Fig. 5 and 8, the automatic calibration results of StopSignV2 are better than those of StopSignV1, for both camera 5 and 6. The reason is that more images were used in the calibration, the system will be more robust to some "bad matches" (e.g. the estimated lines deviate from the correct positions). Thus, the manual work is only useful when there are not many input images, which is rare in practice.
Notice that the uncompressed StopSignV2 data should contain richer information than the compressed StopSignV1. However, experiments show that, when sliced into subsets of similar sizes as StopSignV1, the relative errors of StopSignV2 are not significantly smaller than those of the former. It is, therefore, reasonable to conclude that for real applications, large number of images are preferred for accurate autocalibration using our system.

E. Characterizing the Error
In this section, we present to the readers intuitively the effect of the relative error using a simple example of distance measurement. Suppose the principal point of a camera is fixed at the center of the image plane. A 3D point P = (X, Y, Z) T in the camera coordinates is projected to a pixel p = (u, v) T in the image plane. The pinhole camera model gives the relation: where f x , f y are the focal lengths estimated using our system. If an algorithm measures the object-to-camera distance Z from images captured by a camera calibrated using our system, the measurement error of Z (assuming the algorithm itself doesn't introduce any error) is equal to the relative error of the estimated focal lengths fx , fy . Take StopSignV2-Camera5 as an example (which achieved a 5% relative error), an object at 50m away will be estimated between 47.5 and 52.5m. This error range is acceptable for reacting safely in urban areas with a 25mph speed limit.
V. CONCLUSION Our automatic system for intrinsic camera calibration using stop signs shows to have good performances in urban scenarios, with a convergence trend to solutions comparable with the chessboard calibration when the image number increases and a Kalman filter is applied. The system is more robust as the number of images is higher. The ability of continuously and accurately tracking the intrinsic parameters when driving on the road enables the long-term operations of a camera system for large-scale autonomous driving applications. It is also of interest to mention that chessboard calibration is more suitable to be considered as the baseline instead of the ground-truth. Experiments show that there were 1-2% variations in the chessboard calibration, which means the relative errors of our system may be smaller. In addition to the good performance, the system framework has the advantage of being easily adapted to using other traffic signs or landmarks. This allows us to create an autocalibration toolbox, which will greatly benefit the whole autonomous driving community.
For future work, we will replace the HSV color space segmentation in the contour points detection with a better method (e.g. semantic segmentation), to improve the contour points detection at night. Also, we will introduce the Levenberg-Marquardt algorithm to refine the calibration results. Experiments will be carried out to compare the refined results with the current results.
ACKNOWLEDGMENT Many thanks to Hengyuan Zhang for chessboard calibration, Andres Gutierrez for data pre-processing and all AVL lab members for the insightful discussions.