{'fraction_non_alphanumeric': 0.016670110904456845, 'fraction_numerical': 0.0037886615691947374, 'mean_word_length': 4.957324579400903, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Importance of Camera Calibration and Distortion Correction to Obtain Measurements with Video Surveillance Systems

Video surveillance systems are commonly used as important sources of quantitative information but from the acquired images it is possible to obtain a large amount of metric information. Yet, different methodological issues must be considered in order to perform accurate measurements using images. The most important one is the camera calibration, which is the estimation of the parameters defining the camera model. One of the most used camera calibration method is the Zhang's method, that allows the estimation of the linear parameters of the camera model. This method is very diffused as it requires a simple setup and it allows to calibrate cameras using a simple and fast procedure, but it does not consider lenses distortions, that must be taken into account with short focal lenses, commonly used in video surveillance systems. In order to perform accurate measurements, the linear camera model and the Zhang's method are improved in order to take nonlinear parameters into account and compensate the distortion contribute. In this paper we first describe the pinhole camera model that considers cameras as central projection systems. After a brief introduction to the camera calibration process and in particular the Zhang's method, we give a description of the different types of lens distortions and the techniques used for the distortion compensation. At the end some numerical example are shown in order to demonstrate the importance of the distortion compensation to obtain accurate measurements.


Introduction
Video surveillance systems are even more often installed especially where a continuous monitoring is necessary. They are commonly used to monitoring highways, banks, shopping centers, schools and so on. Some years ago the information acquired from video surveillance systems was only qualitative but lately the idea of using these systems as important sources of quantitative information is even more diffused. In fact with an adequate system configuration it is possible to add to qualitative information a large amount of metric information, that can be used to make measurement on objects in the scene, or calculate distances, or even automatically recognize faces or obtain motion speeds and directions. In order to obtain quantitative information from images it is necessary to calibrate the camera. Camera calibration is the process of estimating all the camera model parameters and it can be divided in to main steps: the estimation of the parameters defining the linear model and the estimation of the nonlinear distortion parameters. The most appropriate linear model for video surveillance camera is the pinhole camera model [2], that is described in the chapter 2. The camera calibration process is then described in chapter 3 and in particular the Zhang's method [2]. This broadly diffused technique allows to calibrate cameras using a simple and fast procedure and it leads to accurate results. It is then the method chosen to perform camera calibration in the experimental studies of chapter 5. The Zhang's method only considers the linear camera model, so it neglects the nonlinear parameters that describe the lens distortions. This simplification can be done only in case of high quality optics with long focal lengths. As optical systems used in video surveillance are characterized by poor quality and short focal lengths, it is necessary to consider and compensate the optical distortions. In this paper we describe the different types of distortion and the procedure to compensate them. Then a numerical example is shown in order to demonstrate the importance of distortion compensation to perform accurate measures.

Camera model
The image formation process consists in the mapping between the three-dimensional world and the twodimensional image. The camera can be described by some different models depending on the camera type. In case of video surveillance systems the most appropriate and simple model is the pinhole camera. This model describes the camera as a central projection system. The camera is considered as a box with a very small hole through which the light coming from the scene passes. On the opposite face of the box an upside down image is projected. The camera aperture is described as a point and no lenses are used to focus the light. Therefore this model does not consider distortions of lenses, blurring of unfocused objects and the concept of depth of field. Geometry describing the pinhole camera model is shown in figure 2. The world points are projected with respect to the camera (or optical) center on the image plane. The distance between the camera center and the image plane is called focal length. The set of camera parameters consists of two categories: the internal parameters and the external parameters. The former are these parameters that describe the physical model of the camera (focal length, coordinates of the optical center, pixels size, etc…), the latter only describe the pose (position and orientation) of the camera with respect to a reference system. The pinhole camera model doesn't consider the lenses distortions. As we describe more in detail in chapter 4 with video surveillance systems it is worth complicating the camera model in order to consider the distortions and to perform more accurate measurements.

Camera calibration
Camera calibration is the process of estimating the parameters that compose the camera model (internal and external parameters). In scientific literature there are different approaches to camera calibration based on different techniques [3] [4]. One of the most used techniques is the Zhang's method. This technique only requires the camera to observe a planar pattern (for example the one in the figure 3), which must be captured at least in three different orientations. Either the camera or the planar pattern can be freely moved and the motion does not need to be known. The calibration setup is very simple and the procedure is easy and fast. In the experimental studies in chapter 5 we use a particular method based on Levenberg-Marquardt algorithm [5].

Distortions
As described in chapter 2 the pinhole camera does not consider the optical aberrations. An optical aberration is the difference between the image that is created and the image that was desired. Aberrations are divided in two classes: monochromatic and chromatic. Monochromatic aberrations are caused by the geometry of the lens and occur when light is reflected and it is refracted (they appear even when using monochromatic light). Chromatic aberrations are caused by dispersion, the variation of a lens's refractive index with wavelength. They do not appear when monochromatic light is used and thus can be neglected in video surveillance systems. Among all the monochromatic aberrations the lens distortions (or image distortions) are the most common. An image distortion is the deviation from rectilinear projection (the projection in which straight lines in the scene remain straight in the image), due to the non linearity of the lens. Although distortion can be irregular or follow many patterns, the most commonly encountered distortions are radially symmetric (or approximately so), arising from the symmetry of the lens. The radial distortion can usually be classified as barrel distortions and pincushion distortions. In the first type of distortion the image magnification decreases with distance from the optical axis. Images subject to this distortion appears as mapped around a sphere (or a barrel). An example of image with this type of distortion is shown in figure 4a. In pincushion distortion the image magnification increases with the distance from the optical axis. The apparent effect is that lines that do not go through the center of the image are bowed inwards. An example of this type of distortion is shown in figure 4b. Less common is the mixture of both types, called mustache (or complex) distortion (figure 4c). Mathematically barrel and pincushion distortion increase as the square of the distance from the center and can be modeled in different ways. It is possible to consider another component of distortion where the image magnification increases moving perpendicularly to the optical axis. This type of distortion is called tangential distortion and most of times can be neglected compared to the radial distortion. One of the most used ways of modeling distortion is described in [6]. The radial distortions depend typically on the focal length. Usually the barrel distortion occurs in short focal length and wide-angle lenses. For example fisheye lenses, which take hemispherical views, utilize barrel distortion as a way of map an infinitely wide object plane into a finite image area. The pincushion distortion occurs with long focal length. In video surveillance systems a short focal length is often used because of the necessity to have a large view of the scene. In these particular systems the occurrence of barrel distortion is very common. To perform any kind of measure on images from video surveillance system it's essential to consider the occurrence of lens distortions and compensate them. The distortion can be correct with a post processing analysis. To do so it is important that the camera model used during the calibration process has taken the distortion parameter into account. To perform a good calibration it is not necessary to consider every type of distortions, but it is important to consider only the relevant ones. Taking into account some kind of distortions that are not necessary can make the camera model too complex and limit the accuracy of the calibration process. If the distortion parameters are correctly estimated, it is possible to eliminate the image distortion by warping it with a reverse distortion. This means determining which distorted pixel corresponds to each undistorted pixel. This can be done on the image, creating a new image that does not show distortion and making measurements on that, or mathematically directly on the measure, canceling the distortion contribution with mathematical equations. These processes are non-trivial due to the non linearity of the equations that describe the distortions. There are different algorithms and software that allow canceling the distortion [7].

Experimental studies
A first example of image subjects to distortion is shown in figure 5a. The image shows the view of a corridor from a camera that is subject to significant radial distortion (barrel distortion). It is easy to see how the jambs of the doors are not straight lines, especially away from the center of the image. The distortion in the image has been compensated using an algorithm from HALCON libraries. The result is shown in figure 4b, where the jambs appear straight. Every measure done on the distorted image without considering the distortion will be wrong. To demonstrate the importance of distortion rectification we show a numerical example. The figure 6a shows a planar pattern used to perform the calibration. This image is acquired from a device with wideangle lens and subject to very strong barrel distortion. This is clear looking for example at the edges of the table on which the pattern is placed, which appear very non-straight. The geometry of the calibration pattern is known with high accuracy. It's a square of 200X200 mm with 49 blobs (7X7) 25 mm apart. First of all a set of images is acquired to perform the camera calibration using the HALCON procedure, that is a modification of the Zhang's algorithm. With the calibration process it's possible to estimate all the parameter (intrinsic and extrinsic) of the camera and the parameters that describe the distortions. We performed two different calibrations: one considering the distortions and one without considering them. We call b1 the blob in the lower left of the pattern, b2 in the higher left, b3 in the higher right and b4 in the lower right of the calibration pattern. We call d1 the distance between b1 and b2, d2 the distance between b2 and b3, d3 between b3 and b4 and finally d4 the distance between b4 and b1. We performed measurements on d1, d2, d3 and d4 in three different cases. The first measurements are performed on the image distorted using the calibration parameters that consider the distortion. In this case distortions are compensated mathematically. In the second approach we created a new image, shown in figure 6b, where the distortions have been erased using calibration parameters that consider distortions. On this image we can perform measurements without considering distortion parameters and the results must be similar to the first method. A third approach is performed to show that not considering the distortions leads to non-negligible errors. In this case the measurements are performed on the distorted image using calibration parameters that don't consider distortions. The results are shown in b) Figure 6. Image of the calibration pattern measured. a) Image with distortion; b) image with the distortion corrected. From these data it's not difficult to understand that neglecting distortions produce a considerable error in measurements. In this case the camera is subjected to high barrel distortions and the distances measured without considering distortions are really different from the real distances and make no sense. This is not true for all the cameras. There are situations in which neglecting distortions does not lead to wrong results.

Conclusions
In order to perform measurement on images from video surveillance system it is necessary to calibrate the camera. The calibration is the estimation of the parameters that compose the camera model. In order to increase the accuracy of the measurements, it often worth to add to the model some parameters that describe the distortions caused by the lens. Using high quality and long focal lenses it is possible to neglect them without losing accuracy. In video surveillance systems a short focal length is often used because of the necessity to have a large view of the scene. In this particular situation the occurrence of barrel distortion is very common and considering them can increase considerably the accuracy of the measurements. In this paper we demonstrated how the measurements performed on an image with significant distortions without compensate them are inaccurate and would lead to wrong results.