{'fraction_non_alphanumeric': 0.02734405960212412, 'fraction_numerical': 0.007688039946104462, 'mean_word_length': 4.9800947867298575, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Coding of distortion-corrected fisheye video sequences using H.265/HEVC

Images and videos captured by fisheye cameras exhibit strong radial distortions due to their large field of view. Conventional intra-frame as well as inter-frame prediction techniques as employed in hybrid video coding schemes are not designed to cope with such distortions, however. So far, captured fish-eye data has been coded and stored without consideration to any loss in efficiency resulting from radial distortion. This paper investigates the effects on the coding efficiency when applying distortion correction as a pre-processing step as opposed to the state-of-the-art method of post-processing. Both methods make use of the latest video coding standard H.265/HEVC and are compared with regard to objective as well as subjective video quality. It is shown that a maximum PSNR gain of 1.91 dB for intra-frame and 1.37 dB for inter-frame coding is achieved when using the pre-processing method. Average gains amount to 1.16 dB and 0.95 dB for intra-frame and inter-frame coding, respectively.


INTRODUCTION
In many application scenarios, extreme camera optics that are quite different from conventional rectilinear lenses are made use of. Such scenarios include surveillance, automotive, or outdoor applications, where typically ultra wide-angle lenses (e. g., fisheye lenses) or catadioptric cameras consisting of both lenses and mirrors are employed to capture extensive fields of view of up to and above 180 degrees. Fields of view of this dimension have the distinct advantage of being able to preserve all the visual content of a hemisphere and map it onto the image plane. While this is indeed useful for surveillance and other applications, the mapping leads to the introduction of strong radial distortions in the final images and videos. This in turn leads to the captured material exhibiting properties that are not taken into consideration by conventional hybrid video codecs such as the recently standardized H.265/HEVC [1,2] or its predecessors.
Hybrid video codecs are block-based and contain procedures for predicting the current block either spatially from the current video frame (intra-frame prediction) or temporally from preceding frames (inter-frame prediction). When using inter-frame prediction, translational motion between frames can be estimated, compensated, and thus exploited for efficiently coding the current block. Motion estimation and compensation techniques realizing this inter-frame prediction are typically designed for video material with rectilinear properties, and do not take into account that translational motion in radially distorted video sequences does not preserve shapes, which in turn impairs the block-matching procedure. Similarly, intra-frame prediction works very efficiently on images containing clear lines and edges. As radially distorted images contain arcs instead of lines, efficiency decreases accordingly.
In an application scenario where it is necessary to correct the radial distortion, the distortion correction is conventionally done after coding. While the encoding step is typically part of the capturing and storing of the data, distortion correction is a post-processing step after decoding and thus not integrated into the camera. Considering the above-mentioned inefficiency of intra-frame and inter-frame prediction in the presence of radial distortion, it seems prudent to employ the correction as a pre-processing step instead.
This paper focuses on radial distortion and its effects on video coding. It suggests a new processing order for the coding and distortion-correction of fisheye video material and also provides a comparison to the conventional processing chain. The paper is organized as follows. Section 2 deals briefly with the acquisition of the parameters required for distortion correction. In section 3, the two different methods of coding and distortion correcting fisheye video sequences are described, the second of which is our proposed new processing order. Section 4 discusses the simulation setup and results, and section 5 gives a conclusion of this paper.

CAMERA CALIBRATION
Due to the large field of view, images and videos captured by fisheye cameras exhibit strong radial distortions. Fig. 1 shows some example frames of such distorted video material. In many cases, it may be desirable to correct the distortion to obtain rectilinear image properties where straight lines in the scene actually appear as straight lines in the image -just as if they were taken by a conventional perspective camera. To that end, it is necessary to perform a camera calibration to determine the camera parameters.
For our purposes, we used a completely automatic technique that only requires a few calibration images showing a checkerboard pattern at different angles and positions [3]. Part of the calibration procedure is the estimation of the nonlinear imaging function which effectively relates a point in the sensor plane to a scene point. Part of this imaging function is described by the 4th-order polynomial is a radially symmetric function of the Euclidean distance ρ to the image center and the coefficients a n denote the intrinsic parameters, which the calibration procedure tries to approximate.
With the knowledge of the intrinsic parameters, the distortion can be corrected. The used toolbox allows adjusting the viewing distance via a zoom factor sf . The greater this parameter is chosen, the higher the amount of image content that is preserved. However, the perspective distortion becomes more pronounced. Fig. 2 shows a distortion-corrected example frame of video1 for two different distance settings, near (DCOR5) and far (DCOR9).
Further elaboration on the calibration procedure as well as the distortion correction is outside the scope of this paper. Details are found in [3] and also in the tutorial accompanying the used toolbox [4]. More general information on projective geometry and radial distortion and its correction may be found in [5,6,7], for example.

METHODS FOR DISTORTION-CORRECTION AND CODING
There are basically two possibilities to code and correct distorted videos. The conventional way is to employ radial distortion correction (DCOR) as a post-processing step after decoding (postDCOR). The other possibility is to perform the correction first and then encode the corrected videos (pre-DCOR). Fig. 3 compares the two processing chains which are further described in the following.

Post-correction of distorted video sequences
Typically, images and video sequences are captured by, for example, on-board car or outdoor cameras and directly encoded (e. g., in MotionJPEG or H.264/AVC [8] format) for efficient storage. The top half of Fig. 3 depicts this stateof-the-art processing order. If a user requires the data to be rectilinear, the correction is done after the data has been transmitted from the camera to some other device and decoded for further post-processing steps. As the original, distorted data is encoded, the radial distortion of the images is not taken into consideration during compression, and the coding efficiency may deteriorate accordingly.

Distortion correction as a pre-processing step
The bottom half of Fig. 3 shows our proposed processing order. In contrast to the post-correction case, we propose the distortion correction to be done as a pre-processing step before encoding the captured data. That way, the images and videos to be coded are rectilinear and thus perfectly suited for both temporal block-matching and spatial angular prediction. Of course, this processing order requires the camera parameters to be already available, i. e., the calibration has to be done beforehand as well. As there are no deteriorating effects caused by radial distortion, the coding efficiency is expected to increase for this pre-correction method. This is shown to be true in the next section.

SIMULATION RESULTS
For our simulations, we tested four traffic video sequences captured by fisheye car cameras. Each sequence comprises 30 frames and only the luminance component of the raw YUV sequences was considered. Example frames are depicted in Fig. 1. For camera calibration and distortion correction, the OCamCalibToolbox by Davide Scaramuzza [3,4] was used. The distortion-corrected sequences were generated for three different viewing distances: near, medium far, and far, i. e., sf ∈ {5, 7, 9}, also denoted as DCOR5, DCOR7, and DCOR9 (for a visual example, cf. Fig. 2). As a hybrid video codec, the reference software HM-11.0 [9] of H.265/HEVC was selected. The three Main profile configurations Intra, Low Delay, and Random Access were tested for five quantization parameters ranging between 24 and 40. The following two subsections discuss the results of the rate-distortion analysis and perceptual quality evaluation, respectively.

Rate-Distortion Analysis
In terms of objective video quality, the luminance PSNR (PSNR Y) was evaluated for different quantization parameters. For this purpose, the output sequences of both the postDCOR and the preDCOR processing chain were compared to the distortion-corrected versions of the original video sequences as indicated by the dashed line in Fig. 3. Furthermore, the bitrate in bits per pixel (bpp) was determined. Fig. 4 shows indicative rate-distortion results for one of the four test sequences. From left to right, the curves for the Intra, Low Delay, and Random Access mode are depicted, respectively. The results for the remaining three video sequences are similar to the ones shown.
Comparing each dashed line to the corresponding solid one yields a PSNR Y increase of up to 2 dB, corresponding to bitrate savings of about 10 to 20 %. This holds true for all three distortion-corrected versions of the sequences. To create average PSNR results over all tested quantization parameters, the Bjøntegaard quality metric (BD-PSNR) [10] was employed. Table 1 provides the BD-PSNR results in the form of PSNR differences for all four video sequences. The ∆PSNR values were obtained using the postDCOR PSNR curves as reference curves. Positive values hence denote a gain, meaning that a quality improvement is achieved when  Table 2 summarizes the average relative bitrate differences for the three modes. Negative values denote a bitrate reduction, so that again, the preDCOR method always performs better than the postDCOR method for the tested sequences.

Perceptual Quality Evaluation
For further evaluation and comparison of the visual quality, the structural similarity (SSIM) index [11] was used as a perceptual quality metric. The distortion-corrected original videos served again as reference sequences for both the post-DCOR and the preDCOR chain. In order to provide a fair comparison, the SSIM evaluation was done equivalently to the rate-distortion analysis, i. e., the averaging is also based on the Bjøntegaard method. Like before, the postDCOR results served as the reference results to determine whether a quality gain or loss is achieved. In addition to the above, a visual comparison of the two output sequences was made to confirm the visual quality improvements implied by the PSNR and SSIM results. Fig. 5 shows an example for intra-frame (top) and inter-frame coding (bottom) using a quantization parameter of 32. The reference frame (DCOR5, left) is compared to the two processing methods (middle and right). As can be seen, straight lines are better reconstructed with the preDCOR method.

CONCLUSION
In this paper, we compared two different processing orders for the coding and distortion correction of radially distorted video sequences. The conventional order that employs distortion correction as a post-processing step was compared to our proposed order of employing distortion correction as a pre-processing step. Evaluation results showed that coding gains as well as visual quality improvements are achieved for the proposed method throughout all tests conducted. We conclude that radial distortion and compensation thereof is a promising means for improving the coding efficiency of hybrid video codecs such as H.265/HEVC.
As there exists no prior work in that regard, future work will include further investigation and analysis of the influence of radial distortion on intra-frame and inter-frame prediction in hybrid video coding and make an effort to exploit the distortion properties for more efficient coding techniques. Similar to the rotational motion estimation described in [12], radial distortion correction could be incorporated into H.265/HEVC as a compensation procedure to that end.