{'fraction_non_alphanumeric': 0.03720207677527493, 'fraction_numerical': 0.010710927599035198, 'mean_word_length': 4.521896162528217, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robust Seatbelt Detection and Usage Recognition for Driver Monitoring Systems

Wearing a seatbelt appropriately while driving can reduce serious crash-related injuries or deaths by about half. However, current seatbelt reminder system has multiple shortcomings, such as can be easily fooled by a"Seatbelt Warning Stopper", and cannot recognize incorrect usages for example seating in front of a buckled seatbelt or wearing a seatbelt under the arm. General seatbelt usage recognition has many challenges, to name a few, lacking of color information in Infrared (IR) cameras, strong distortion caused by wide Field of View (FoV) fisheye lens, low contrast between belt and its background, occlusions caused by hands or hair, and imaging blurry. In this paper, we introduce a novel general seatbelt detection and usage recognition framework to resolve the above challenges. Our method consists of three components: a local predictor, a global assembler, and a shape modeling process. Our approach can be applied to the driver in the Driver Monitoring System (DMS) or general passengers in the Occupant Monitoring System (OMS) for various camera modalities. Experiment results on both DMS and OMS are provided to demonstrate the accuracy and robustness of the proposed approach.


Introduction
Wearing a seatbelt appropriately while driving can reduce serious crash-related injuries and deaths by about half, and in 2016 alone seatbelts have saved 15,000 lives, making it the greatest protection for passengers especially when combined with air bags (CDC 2021a) (CDC 2021b). Current seatbelt reminder system has several shortcomings. First, it can be easily fooled by a "Seatbelt Warning Stopper" (Amazon 2021), a device made with a small metal that fits into the buckle as if a real seatbelt is fastened. Second, incorrect seatbelt usage, such as seating in front of a buckled seatbelt or wearing a seatbelt under the arm, which is also very harmful when crash happens (Intas and Stergiannis 2010), cannot be detected. Third, unnecessary false alarm can happen if pets or bags are put on the passenger seats. Fourth, Current reminder system is not enforced for the back passengers, and parents get distracted while driving for frequently checking whether their children have unbuckled up their seatbelts, e.g. for picking up a toy.
Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Accurate and robust general seatbelt detecting and usage recognizing is a hard problem due to many challenges.
• Seatbelt images may have color, e.g. Fig. 1a, or be gray, e.g. Fig. 1b; • Super-wide Field of View (FoV) fisheye lens may be used, within which the belt is a curve as shown in Fig.  1a, while for regular FoV lens, the belt is more like a straight line, as shown in Fig. 1c; • The cloth and seatbelt's color may be similar, resulting in low contrast image, e.g. Fig. 1b; • Partial of the seatbelt might be occluded by objects, e.g. hands in Fig. 1a; • Seatbelt can be stretched into to an irregular shape, an extreme case shown in Fig. 1b; • Image may be blurry due to object movement or vehicle vibration, e.g. Fig. 1b.
In this paper, we propose a novel seatbelt detection and usage recognition framework to solve the above-mentioned challenges. There are three components within the framework, whose brief descriptions are shown below.
I. Local predictor: Scanning through the input image and predicting for each local patch window whether it is part of a seatbelt II. Global assembler: Assembling the positive seatbelt candidates in step I by removing false positive as much as possible utilizing various attributes, such as shape, location, etc. III. Shape modeling: Building the geometric seatbelt shape model utilizing the filtered candidates from step II, and mapping it on top of the original image We summarize our contributions as follows.
(1) A new seatbelt detection and usage recognition framework that can robustly detect one or more seatbelts for both regular FoV or fisheye images. (2) A new local seatbelt patch predictor that provides binary classification for any small patch window. (3) A global assembling algorithm that can effectively remove false positive seatbelt patch candidates. (4) A high order polynomial curve based seatbelt shape modeling functionality that localizes the seatbelt even when occlusion exists. The rest of the paper is organized as follows. First previous work on seatbelt detection and related topics are reviewed. Then we explain our methodology and technical details. After that we present our experiment results on various contexts. Finally we conclude our paper and provide possible future work.

Related Work
According to WHO (WHO 2009), more than 1.2 million people die yearly in vehicles accidents, but they would have 40% -50% probability of survival if they used seatbelts properly. In addition, seatbelt can reduces serious crashrelated injuries and deaths by about half via reducing the force of secondary impacts with interior strike hazards, saving more than 10K people per year in US only (CDC 2021b). Building the capacity of detecting seatbelt and classifying its usage is not only an interesting research topic, but also widely pursued by auto industrial especially by the brands that are sensitive to safety.
Multiple researchers focus on detecting whether the drivers are wearing seatbelts or not via external traffic monitoring system for law enforcement purpose (Qin et al. 2014) (Hosam 2020) (Yang, Zang, and Liu 2020) (Artan et al. 2015), where one or multiple cameras are installed on the road infrastructure such as near traffic lights. Such systems mainly focus on the seatbelt status of the driver, but not all the occupants inside the vehicle due to the limitation of the camera-vehicle relationship. In this paper, we are mainly interested in seatbelt detection and usage recognition where cameras are installed inside a vehicle.
Fisheye or omnidirectional cameras are widely used for applications where wide Field of View (FoV) representation of the surrounding scenes are needed (Hu, Zhu, and Zhang 2014) (Hu et al. 2017) (Hu et al. 2018). Some researchers use salient gradient feature , line-detection (Guo et al. 2011) or convolutional neural network (CNN) (Zhou, Chen, and Wang 2017) (Chun et al. 2019) (Kashevnik et al. 2020) for regular Field of View (Fov) camera based seatbelt detection. These methods, however, cannot extend to Occupant Monitoring System (OMS) where fisheye or super wide Field of View lens are used and images are extremely distorted. The image of a straight line in the physical world, such as an edge of the belt is no longer straight in a fisheye image, thus simple edge or line detection based ap-proaches fail.
In order to recognize the seatbelt usage beyond just detecting the seatbelt is on or off, such as whether the seatbelt is fastened over the shoulder or under the arm, precise seatbelt curve locations are needed. Curve fitting with limited point candidates is an extensively studied topic (Levy 1959) (Guest and Guest 2012). In our approach, we adopt a polynomial curve based fitting method for fast detection. Since the input fitting data are usually noisy with outliers e.g. due to false positive candidates, RANdom SAmpling and Consensus (RANSAC) (Fischler and Bolles 1981) algorithm is used as well for removing the outliers.
Seatbelts can have many types, such as 2 points, 3 points, or 4 to 6 points belts (e.g. seatbelts for child seat or racing cars), even though we believe our methodology can extend to all these types, in this paper, we only focus on the most popular Y shape 3 points seatbelts widely used in the auto industrial. For a given car model, the 3 anchor points of a seatbelt are roughly fixed, though some anchor such as the top point may have some degree of freedom.
Camera calibration is a process to estimate a camera's intrinsic parameters (such as focal length, distortion parameters, etc.) and extrinsic parameters (such as rotation and translation) that is necessary for many vision-based solutions (Li, Hu, and Geng 2011) (Ren and Hu 2021). If a DMS or OMS camera is installed, its intrinsic and extrinsic parameters are fixed, so camera calibration such as Zhang's method (Zhang 2000) can be used for determining the camera's position relative to the vehicle. Once the cameravehicle geometric relationship is determined, we can further estimate a seatbelt anchor's location in an seatbelt image using Perspective-n-Points (PnP) algorithm (Moreno-Noguer, Lepetit, and Fua 2007).

Methodology
We design a bottom-up framework to tell for any given input image from a DMS or OMS system, whether the seatbelt is detected or not for a specific seat, and if detected, whether it is correctly fastened, e.g. not under the arm. First, from the lowest level we design a process to roughly predict for any input pixel whether it is part of a seatbelt pixel set, using itself and its neighborhood pixels' attributes. Then, we filter out the false positive candidates and only assemble the qualified ones utilizing global seatbelt features Figure 2: Seatbelt patch geometry example such as geometric location, smoothness or intensity range. Finally, we utilize the seatbelt anchors' location and the candidates survived in previous step to model the shape of the seatbelt, which is further visualized by overlapping the shape on top of the original image. We summarize the three steps in our framework as the local predictor, the globally assembler and the shape modeling.
In this paper, we will use a single belt within one seatbelt image as an example, though our proposed approach also applies even if there are multiple belts within a image.

Local predictor
The local predictor predicts whether a pixel belongs to a seatbelt or it is a background pixel using the pixel itself and its neighbors' information.
The predictor outputs 1 or 0 representing seatbelt or nonseatbelt respectively. Due to noise, this binary classification may have four possible cases as shown below.
• True Positive (TP): a pixel is part of a seatbelt, and the predictor returns 1; • False Positive (FP): a pixel is not part of a seatbelt, but the predictor returns 1; • False Negative (FN): a pixel is part of a seatbelt, but the predictor returns 0; and • True Negative (TN): a pixel is not part of a seatbelt, and the predictor returns 0.
In this step, we adjust the predicting criteria to reduce the false negative (case FN) as much as possible, even if at the cost of increasing false positive (case FP). In other word, high recall is preferred over high precision.
Notations Denote p(x, y) an arbitrary pixel locates at y th row and x th column within an image Img as shown in Fig.  2. The image width is W , height H , and it has K channels, where K = 3 (or 4) for color images, and K = 1 for gray images.
We define a square around p(x, y) as its neighborhood set. Assume the length of the square is L = 2k + 1 pixels, where k = 1,2,3. . . , the set S can be formulated as: Since seatbelt can extend along different directions in an image, as shown in Fig. 1a and Fig. 1b, isotropic predicting, i.e., checking along different directional angle for each pixel, is necessary. The local predictor returns binary prediction result vectorr(x, y) for each pixel as defined in Equ. 2.
where D is the total number of directions, θ 0 , θ 1 , . . ., θ D are evenly spaced angles within the range of [0, π). Define where d = 0, 1, . . . , D − 1. The criteria function f criteria (x, y, θ d ) determines for any pixel p(x, y) within image Img and a specific direction θ d , the patch image generated using p and its neighborhood S at this direction is a seatbelt component or not. We will describe with more details about the criteria function later in this section. In addition we define the local predictor f predictor (x, y) as a scale function ofr(x, y): where w(θ d ), d = 0, 1, . . . , D − 1 are weights assigned to different directions.
Observations Assume we have fetched a seatbelt patch for an arbitrary pixel together with its L × L neighbors in a targeted direction (patch direction), as shown in Fig. 2, we observe multiple properties.
• Observation (I): structured edges. Seatbelt edges generate two usually parallel lines in the patch direction even in fisheye images, if the patch is small enough.
• Observation (II): intensity or saturation range. Seatbelt usually comes with black, grey, or tan color. Though pixel values change in some context such as illumination changes, we still can exclude extreme values experimentally.
• Observation (III): surface smoothness. Standard seatbelts usually have similar texture along the belts thus the pixels within seatbelt region will have limited variety in terms of smoothness.
Denote f structure (x, y, θ i ), f intensity (x, y, θ i ), and f smoothness (x, y, θ i ) as binary functions modeling above three observations, then: Criteria (I): structure We will first discuss how to identify and localize candidate seatbelt edges, then we will talk about how to use it to distinguish seatbelt pixel from nonseatbelt noise. Each seatbelt patch, i.e. a candidate pixel plus its neighbors in a square, has a patch direction by definition, and for each patch, we only check whether there is a seatbelt parallel structure along the patch direction.
As shown in Fig. 2, for an arbitrary patch centered at pixel p(x, y) with patch direction θ, we retrieve its L * L pixel values from image Img using Equ. 6.
where i, j = 1,2,. . . ,L. Fig. 3a and Fig. 3e show two sample seatbelt and nonseatbelt patches, where the seatbelt patch has a parallel structure, i.e. two parallel vertical lines, along patch direction but the non-seatbelt patch does not. If we localize the positions of the structure, we have detected the seatbelt boundary within this patch.
We utilize 2-dimensional gradient operation to capture the magnitude of sudden intensity change, and accumulate along patch direction to obtain the patch 1D curve representation. Denote f patch (x, y, θ, j) as the resulting curve representation after this projection.
(7) where j is the variable indexing patch columns, i indexing rows, and f (x, y, θ, i, j) is patch intensity function. Fig. 3b and Fig. 3f show the patch images after gradient operations, We can further smooth the curves e.g. using Savtzsky-Golay algorithm (Gorry 1990) to get a smoothed curve f patch (x, y, θ, j) as shown in Fig. 3d and Fig. 3h.
Observe that a seatbelt patch with structured edges will have two peaks as shown in Fig. 4, while non-seatbelt patch will have random curves. We characterize this observation with two variables: the inter-edge distance d edges and peak value difference d peaks . f structure (x, y, θ i ) is therefore defined in Equ. 8.
To calculate d edges and d peaks , we need to identify the two peak values ( peak lef t and peak right ), and their respective indexes (idx lef t and idx right ) for a given curve. The problem is converted to a one-dimensional twoclasses classification problem via finding the optimal cutting position idx optimal among all the possible idx, where idx = 1, 2, . . . , L, by maximizing the inter-class distance. where, After we figure out idx optimal , idx lef t and idx right are calculated by searching for the maximum elements position in [1, idx optimal ] and (idx optimal , L]. Their respective function values are therefore peak lef t and peak right . Criteria (II): intensity Seatbelt pixel's intensity varies per seatbelt models, and may change when environmental illumination changes. However, in major Driver Monitoring Systems (DMS), we use active InfraRed (IR) to illuminate and spectral filter to remove external illumination's influence. So for a specific working seatbelt, its intensity distribution has patterns and can be learned from experiments, which we can utilize to assist the local predicting.
Assume δ min and δ max are the lower and upper intensity bounds, and denote weighted intensity for f patch (x, y, θ) as d intensity . where are Gaussian distributed weights assigned to each pixel in the seatbelt patch. Finally we have the definition of f intensity in Equ. 11.
Criteria (III): smoothness The surface of a seatbelt is usually smooth without abrupt intensity changes, while the background are much noisy. Define the intensity variance within a Region of Interest (ROI) ω as d smoothness .
(12) We therefore have the definition of f smoothness in Equ. 13, where ϕ min and ϕ max are experimental thresholds.

Global assembler
The local predictor generates large quantity of seatbelt pixel candidates, including true positive and false positive results.
In this section, we will present how to use global assembler to remove the inaccurate candidates.
Since the seatbelt is always tightened by the built-in springs no matter fastened or not, the location distribution of seatbelt pixels observed via a fixed camera can be roughly determined by the three anchors.
In the following, we will introduce how to utilize this prior knowledge to globally filtering false positive candidates. Furthermore, we will also discuss how to automatic calculate anchors' locations for a specific camera-vehicle configuration.
Global mask generation Once a DMS or OMS camera's pose is configured, i.e. both camera location and its orientation are determined, the seatbelt's footprint in the image will be within a limited area, which we denoted as global mask for this configuration. We will first introduce how to leverage camera calibration and vehicle model to automatically generate the seatbelt global location mask.
Denote the camera is calibrated with intrinsic parameters K, and extrinsic parameters R and T , also the seatbelt has top right anchor A tr (X tr , Y tr , Z tr ) and bottom left anchor A bl (X bl , Y bl , Z bl ) inside the vehicle 3D model. Then the anchors' correspondent coordinates in the image A a (x a , y a ), a = tr, bl can be calculated with Equ. 14.
x a y a 1 where s is a scale factor, and a = tr, bl. According to our experience, majority of location distribution of the seatbelt will fall into an ellipse using the two anchors as ends of the long axis. Denote the major axis distance d major as shown in Equ. 15.
Meanwhile, the minor axis distance can be obtained via analyzing people's seatbelt fastening habits statistically. Assume the minor axis distance is d minor , we have the definition of the global mask set S location shown in Equ. 16.
( 1 2 d minor ) 2 ≤ 1} (16) Note that the (x local , y local ) are defined in ellipse local coordinate system, where the origin O elipse is at the middle point of segment A bl A tr with image coordinate ( xtr+x bl 2 , ytr+y bl 2 ), x-axis pointing from A bl to A tr , y-axis obtained via clockwise rotating x-axis for 90 degrees. Assume the correspondent coordinate of (x local , y local ) in image coordinate is (x global , y global ). where With Equation 16 and 17, we can obtain the location mask S location in the image coordinate. The seatbelt pixel candidates proposed by local predictors outside of the global mask will be filtered out.

Shape modeling
After locally predicting and globally assembling, we obtain, for each seatbelt region of interest, a set of pixel candidates. However, the expected output is a continuous seatbelt curve from anchor to anchor, with which we derive the usage category. In this subsection, we will explain how to model the seatbelt shape using the survived pixel candidates.
Assume the set of candidates from local predictor S predictor and the location mask S location , we define the candidates set for shape modeling S modeling in Equ. 18.
where γ pre is a threshold for local predicting. We model the seatbelt shape using high order polynomial curve. However, the seatbelt may be stretched away while driving, an example of which is shown in Fig. 1b. So curve fitting directly in the image coordinate system is not feasible, since for some points in the x-axis, there are multiple correspondent y values. We solve this problem by first fitting the curve in the local ellipse coordinate system and then convert the result back into the image coordinate system using Equ. 17.
Assume the observed points are (x i , y i ), we model the curve using Equ. 19.
Since there are still outliers in the shape modeling stage, M-estimator sample consensus (MSAC) algorithm (Torr and Zisserman 2000) can be used to remove the outliers.

Experiments
We conduct our experiments for data collected by both Driver Monitoring System (DMS) using regular Field of View (FoV) InfraRed (IR) camera and Occupancy Monitoring System (OMS) using super wide FoV fisheye IR camera.

Usage categorization
We define three seatbelt usage category as shown below.
• seatbelt ON: seatbelt is correctly fastened above a user's shoulder • seatbelt UNDER: seatbelt is fastened in a incorrect way, i.e. under a user's arm • seatbelt OFF: no seatbelt is observed within the image Note that for the seatbelt off case, it can be caused by seating in front of a fastened seatbelt, a "Seatbelt Warning Stopper" is used while the actual seatbelt is off, or the user is forgotten to fasten the seatbelt. There are ways to distinguish from these three sub-cases, e.g. reading the buckle sensor information from the vehicle's Controller Area Network (CAN) signals. However, it is beyond this paper's scope and we in any case send out a warning message when no seatbelt is observed.
To distinguish between "seatbelt ON" and "seatbelt UN-DER", we calculate the angles between image x axis and the line segment of left bottom anchor and points in the seatbelt curve. If a seatbelt is fastened under the arm, the angle is much smaller compared with correctly fastened over the shoulder. Fig. 5a shows seatbelt detection and recognition results for Driver Monitoring System using regular Field of View camera, where the modeled seatbelt is rendered using green curves and its category printed above the curve in red fonts. Fig. 5b shows a seatbelt detection result when the seatbelt is not fastened by a different subject.

DMS results
We test our algorithm with 407 images with seatbelt on and 974 images with seatbelt off from multiple users. The dms.mp4 (https://bit.ly/2ZWddlW) shows sample results.
The recognized seatbelt usage results can be encoded and sent out to the vehicle system, e.g. using CAN, and proper protocol can be defined to reminder the users for correction, such as using seatbelt warning symbols in the dashboard, seatbelt warning sound, or voice reminder.  We test our algorithm with 319 images with seatbelt ON, 312 with seatbelt OFF and 318 with seatbelt UNDER. We also ask the subject to move around to test the robustness of the algorithm.

OMS results
The oms.mp4 (https://bit.ly/3k8dARJ) shows results with various usage category on full video sequences. The method correctly recognizes 97.49% images for seatbelt ON, 100% when seatbelt is off, and 98.11% when seatbelt is fastened under the arm.
Note that seatbelt may be occluded, e.g. by the hand of the user in many frames. However, since our approach is not only just detecting seatbelt pixels, but also modeling the entire curve using the seatbelt candidates, it can recover the occluded part and output a complete seatbelt.

Extreme case
For extreme cases where the seatbelt is stretched away, as shown in Fig.1b, it is very challenging for regular line detection based seatbelt algorithm to work. Fig. 6 shows some intermediate results of our algorithm. The blue points are the assembled seatbelt candidates and the red line is the modeled shape in the ellipse coordinate system. Fig. 7 shows the zoomed in correspondent result after mapping the curve in Fig. 6 to the original image using green dots.

Conclusion
In this paper, a novel seatbelt detection and usage recognition method is proposed with three components: local predictor, global assembler, and shape modeling. The approach utilizes a hybrid of image color/intensity, shape, smoothness, and location to effectively localize a seatbelt from noise background. It works with both regular FOV or fisheye images, color or IR images, and for different categories (such as seatbelt ON, OFF, UNDER, or AWAY). Experiment results are shown to demonstrate its effectiveness and robustness under various conditions.
In the future, we will extend our experiments to more vehicle models and more camera-vehicle configurations.