{'fraction_non_alphanumeric': 0.027773102171351625, 'fraction_numerical': 0.009930988049149974, 'mean_word_length': 4.978616352201258, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Video Indexing Using Face Appearance and Shot Transition Detection

The possibility to automatically index human faces in videos could lead to a wide range of applications such as automatic video content analysis, data mining, on-demand streaming, etc. Most relevant works in the literature gather full indexing of videos in real scenarios by exploiting additional media features (e.g. audio and text) that are fused with facial appearance information to make the whole frameworks accurate and robust. Anyway, there exist some application contexts where multimedia data are either not available or reliable and for which available solutions are not well suited. This paper tries to explore this challenging research path by introducing a new fully computer vision based video indexing pipeline. The system has been validated and tested in two different typical scenarios where no-multimedia data could be exploited: broadcasted political video documentaries and healthcare therapies sessions about non-verbal skills.


Introduction
Videos represent one of the most important media and the automatic analysis of their contents is one of the most investigated topics in computer science related research areas.
Since human face is an important subject in videos, because it is a unique feature of human beings and is ubiquitous, most of proposed video analysis strategies rely on a match of facial features. Thanks to deep learning based approaches a very high accuracy in face matching and retrieval has been reached even on very large image database [25]. Although images in datasets were acquired with different facial expressions, illumination conditions and occlusions, the exploitation of the above mentioned approaches in real world scenarios is not trivial. The most impressive application works deal with movie actor indexing, but it is worth noting that the prior knowledge of appearance models for a cast [13] and the number of searching clusters (i.e. the number of characters) makes the task easier [1]. Besides, support by existing video captions [27] or other contextual knowledge is required [3]. In addition to the particular application field just mentioned, it is quite common in the literature to find works gathering full indexing of videos in real scenarios by exploiting additional media features (audio [15,24] and text [12,8]) that are fused with facial appearance information to make the whole frameworks accurate and robust. Also the task of speaker diarization in videos has taken benefit from state-of-the-art face clustering [4], spatiotemporal Bayesian fusion [14] and Fisher Linear Discriminant analysis [31] for both audio and video signals. multimedia data are either not available or reliable. Two relevant examples are: 1) the indexing of people in news videos, documentaries or reportage (where audio is not necessarily correlated with images due to the presence of the narrator) and 2) video assisted healthcare (where different subjects, often with cognitive diseases, appear sequentially in front of the camera and their non-verbal behaviours are recorded). These cases require a fully appearance-based video indexing but, unfortunately, to the best of our knowledge, this is a very few investigated research topic.
This paper tries to explore this challenging research path by introducing a new computer vision based video indexing pipeline. The proposed pipeline can detect and re-identify people in each image supplying an automatic subject dependent video indexing that can be exploited for different purposes, e.g. to monitor balanced participation in politic talk shows and news, or to handle electronic records in healthcare systems more easily. It works without using any prior knowledge about the number of relevant persons in the video. Moreover, faces are processed without any constraints about appearance, eyeglasses, beard or hairstyle. A shot transition detection system is also introduced and opportunely integrated in the pipeline in order to improve the results. The system has been validated and tested in two different scenarios. At first, the method has been employed for indexing politicians framed in videos of the Canary Islands Parliament (Spain) sessions. In addition, it was exploited for indexing children in videos acquired during sessions aimed at assessing Autism Spectrum Disorders.
The rest if the paper is organized as follows: relevant works on face indexing are reported in Section 2, while the proposed method is presented in details in Section 3. Experiments and results are showed and discussed in 4; Section 5 concludes the paper with conclusions and future works.

Related Work
The problem of re-identifying the same person under multiple views and different perspectives was initially addressed as a multi-camera tracking problem [19]. For a complete review and the evolution of re-identification works, refer to [38]. The person re-identification became a new and independent research line starting from the work in [16], in which the overall appearance of the individual was specifically modelled to deal with several challenges such as different camera angles and illumination conditions, variation in pose and the rapidly changing appearance of loose or wrinkled clothing. Since that pioneering approach, a plethora of works with the same underlying idea has been proposed. In [11], a segmentation model is used to separate the foreground to whom a computer vision pipeline is applied in order to obtain multi-shot reidentification. In [32], Dynamic Time Warping (DTW), widely used for action recognition, has been employed to re-identify the person. A top-push distance learning model (TDL) has been proposed to enforce the optimization on top-rank matching in re-identification combined with the minimization of intra-class variations to improve performance [35]. Wang et al. [34] introduce a discriminative video ranking model by simultaneously selecting reliable space-time features from video fragments. 3D Histogram of Oriented Gradients (HOG3D) features and optical flow energy profile over image sequence are used to produce a representation designed to generate multiple fragments from unregulated video clips. AdaBoost is applied to both Haarlike features and dominant colour descriptors to achieve the most invariant and discriminative signature in [9].
Several approaches have also been proposed in order to achieve efficient video indexing and retrieval like keyframes texture, edge and motion features [29], temporal mapping [2], or temporal patterns combined with sequence matching techniques [21]. Faces have been automatically annotated by clustering methods with a weighted feature fusion in [7], dealing with colour information, but a training set is needed in order to perform a general-learning (GL) training scheme. Blind separation, i.e. labelling with lack of any prior knowledge, was proposed in [28], but the method worked well only in case of a limited number of participants, relative stable video scene and face images captured in frontal-view. Different clustering metrics have also been exploited to fully automatic indexing people when the total number of people is unknown [6].
Some related works relying only on image analysis limit their scope to the identification of main characters' [17] without achieving the indexing of videos, or scalable face retrieval over large datasets [26]. In [20], the face appearance is joint with body information to enhance the face recognition on videos. Deep learning approaches for face retrieval have also recently been proposed [10,23,33,37].

Proposed Method
The proposed method works as follows: first of all, the video is processed in order to detect the different shots by a frame analysis based on the Kullback-Leibler (KL) divergence. In particular, for each shot, information about first, last frames and duration in frames and seconds is stored. During the same video scanning, facial images are extracted and tracked by a framework of cascaded convolutional neural networks (CNNs). Facial Features are then extracted by a ResNet-50 pre-trained on the VGGFace2 dataset [5] and the cosine distance among feature vectors is computed and used as a metric to guess a match among face belonging to the same person. In order to reduce errors, if faces belongs to the same shot, then threshold is relaxed, whereas a strongest threshold is used in case of matching in different camera shots. A block diagram of the proposed solution is reported in Figure 1. In the next subsections, each algorith-mic step will be detailed.

Shot Transition Detection
Video segmentation into shots allows to characterize them in terms of boundaries and duration, in frames and seconds. The segmentation of the video is carried out following the method proposed by Sánchez-Nielsen et al. [30]. The method is based on the computation of the Kullback-Leibler divergence (D KL ) between every two consecutive frames, frame i and frame i+1 , to assess how similar are their color distribution, P i and P i+1 respectively. The color distribution of each frame is computed as the concatenation of the YCbCr components histogram discretized into 16 bins and concatenated to get one histogram per frame. After that, a normalization stage is done. Under the hypothesis that two frames of the same shot have similar color distributions, shot boundaries are detected when the KL divergence between the color distribution of two consecutive frames D KL (P i ||P i+1 ) is larger than a threshold. Thus, (1) In this work the threshold value, thr, has been set to 0.05.

Multi-Face detection
Since faces can vary in appearance for changes in pose, lighting and occlusions, face detection is a challenging task in real-world applications. In this paper, the results recently proposed in [36] have been used. The aforementioned work relies on a framework of three multi-task cascaded CNNs. In our work, the input image is initially resized to different scales to build an image pyramid which is the input of the cascaded framework. The same fully convolutional network of [36] called Proposal Network (P-Net) is exploited to obtain the candidate windows. Non-maximum suppression (NMS) is then used to collapse highly overlapping boxes. Face localization is further improved by using a bounding box regression module. All candidates are fed to another CNN, called Refine Network (R-Net), which rejects false candidates. Differently from [36], the filtered information of face classification is only extracted and then further processed in our algorithmic pipeline. The configurations of the two CNNs are reported in Figure 2

Facial Feature Extraction
The employed network architecture in this step is the SE-ResNet-50-256D with 30M parameters trained on the VG-GFace2 dataset [5]. The considered architecture contains a the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels [18]. The VGGFace2 dataset contains 3.31 million images of 9131 subjects (identities), with an average of 362.6 images for each subject. Images are downloaded from Google Image Search and have large variations in pose, age, illumination, ethnicity and profession (e.g. actors, athletes, politicians). The whole dataset is split to a training set (including 8631 identites) and a test set (including 500 identites). A face descriptor is obtained from the trained networks as follows: first, the extended bounding box of the face is resized so that the shorter side is 256 pixels; then, the center cropping of the face image is carried out after an enlargement of 20% in each direction, in order to include a piece of background. This process made the informative content compliant with the samples used to train the deep neural network. The outcome of the cropping is used as input to the network. The face descriptor is extracted from the layer adjacent to the classifier layer. This leads to a 256 dimensional descriptor. The vector is finally normalized using L2 norm.

Experiments and Results
Experiments were carried out on two different datasets. The first one, named Dataset #1, contains a collection of videos recorded during debating sessions of the Canary Islands Parliament in Spain. Videos are available at https://www.parcan.es/video. These videos contain a large number of shots and different persons involved in a politic debate. In particular, six videos of this huge dataset have been processed. Three of the selected videos have a duration of about 10 minutes (9m43s, 9m41s and 11m23s respectively), whereas the remaining ones are shorter (about 5m each). Videos were recorded on different days and this choice was made in order to test the system on a large number of persons, with different facial appearance (glasses, hairstyle, beard) and to consider different framing conditions and angles. In Figure 3, some frames extracted from Dataset #1 is reported. As can be observed, videos in this dataset have a number of different camera shots. Each changing in the camera view is totally unrelated to the audio, thus the use of multimedia information would lead the traditional approaches to fail in the identification of the person framed.
The second dataset (Dataset #2) contains two videos recorded during therapeutic sessions for assessing the Autism Spectrum Disorders in children at an healthcare centres located in Alessano, province of Lecce, Italy. The above center offers intervention programs for children with Autism Spectrum Disorders diagnosis and/or other disorders. The two videos frame 17 children (14 boys) with Autism Spectrum Disorder diagnosis, aged 6-13 years (Mean = 8.94; Standard Deviation = 2.41) without cognitive delay. The "L'Adelfia" Ethics Committee gave approval for this study and informed signed consent was obtained from   parents. Videos in this second dataset are longer than in the former. Each video lasts more than two hours. In each video, different children alternate in front of a static web-cam to perform training sessions aiming at improving their skills in understanding and producing facial expressions. Until automatic tools will be available to analyse acquired data [22], after the session the caregiver could require to retrieve images related to a specific child in order to better assess some aspects of the session but, unfortunately, without automatic indexing, the caregiver needs to scroll the whole video with a large wasting of time and to manually identify the frames where single child appears. Unfortunately, traditional indexing approaches fail in long sequences since, also in this application case, there is no correspondence between audio and person framed (often there is no audio at all) since the therapist's voice is the same for different children. From a technical point of view, in these videos, there is only one camera view (the camera was fixed on a tripod) and one child framed at a time. Children alternated in front of the camera to perform their therapeutic training exercises. The children do not speak during most of the time, but they try to accomplish the task that is asked of them, that is, to recognize and produce facial expressions. For privacy issues, images of this dataset cannot be published. Ground truth data for both datasets are reported in Table  1. Table reports, from left to right, the identification label of each video (first column), the number of frames (second column), the number of appearing persons (third column) and the number of different camera shots (fourth column). It is important to underline that a person was considered in the scene only if his face, at least in one frame, was framed with a minimum resolution of 48 × 48 pixels.
In the experimental phase, each video was fully processed using the pipeline described in Section 3. For videos in Dataset #2, the shot transition detector has not been used, manually settings all the frames as belonging to the same shot. The facial patches with and corresponding feature vectors (having 256 items each) were extracted using algorithms explained in Section 3.3, whereas the time bounds of each camera shot were extracted using the methodology described in Section 3.1.
Before carrying out the experimental tests for the whole pipeline for people indexing purposes, a preliminary proof aiming at checking the accuracy of the algorithm implemented for multi-face detection was done. Face detection accuracy was evaluated by annotating 1% of the images in each video and then comparing annotated facial bounding boxes with those automatically labeled by the detector introduced in Section 3.2. In this evaluation step, extracted facial regions that have the Intersection-over-Union (IoU) ratio less than 0.5 to any ground-truth faces were considered as False Positives (FP), whereas extracted facial regions with IoU above 0.5 to a ground truth face were considered as True Positives (TP). As a result of the above, any ground-truth face not having a corresponding detected face was accounted as a False Negative (FN). Table 4 sums up the face detection accuracy according to the aforementioned criteria. From the table it is evident that the face detection worked very well on the considered videos. In particular it is very accurate on videos A1 and A2 since they have a simpler background and only one person at time in the scene. Some faces were not detected by the system due to large occlusions or high facial orientation angles. Some examples are reported in Figure 4. Mostly of false positives correspond to incorrect bounding box positioning that brought to IoU value lower than considered threshold. Following this very satisfactory intermediate evaluation, the whole pipeline was then tested. This final evaluation was carried out by computing the cosine distance between feature vectors pertinent to facial patches extracted from the same video. In Figure 5, a graphical representation of the distances computed on the 12670 facial patches detected in the video P 1 in Table 1 is reported. The figure qualitatively points out how distance values change accordingly to the framed persons. First of all, it is worth noting that blue values follow the diagonal and this means that lowest distance values are properly related to framed persons. Going into details, starting from the top-left corner, the blue (with some shade of light-blue) rectangle represents about the first two minutes of the video in which the President of the Canary Parliament is most of the time framed. The cosine distance is in this case very small since, given the considered colormap, the blue color is associated to the distance values lower than 0.4. After about two minutes, the parliamentary discussion starts and the camera moves on the 5 different persons involved at different zoom levels (approximately from minute 2 to minute 3.30). Then the President is newly framed for a few seconds, and then a new depute is mainly framed till the end of the video. In Figure  6, the facial patches (sub-sampled with a ratio of 1 every 25 extracted) are plotted along time with Y-axis representing the magnified (×1000) cosine distance from the first detected face (i.e. the President of the Parliament). This figure clarify even better that the cosine distance on facial feature vectors is very effective to distinguish between the different framed persons. Concerning the automatic detection of camera shots, it was used only on the parliamentary videos since videos in the second dataset have only one camera shot each. All the camera shots of the videos in Dataset #1 were correctly detected (100% detection rate).
Finally, for assess the whole framework, the following indexing metrics were considered: • True re-identification (TR), also known as true match or true positive: a true re-identification occurs when faces that have the same id label (i.e. low cosine distance) belong to the same person; • True distinction (TD), also known as true non-match or true negative: a true distinction occurs when faces that have different idlabels belong to different persons; • False re-identification (FR), also known as false positive or false match: a false re-identification occurs when faces with the same idlabels belong to different persons; • False distinction (FD), also known as false negative or false non-match: a false distinction occurs when faces that have different idlabels belong to the same person.
In this paper the True re-identification (TR) score has been considered and system's outcomes evaluated when using only cosine distance on facial features extracted by ResNet-50 and by integrating information about camera shot transition detection. In the experimental phase, person identification labels were automatically assigned to each persons according to the order of appearance in the video. This means that the first detected face is identified by id = 1, the second one has id = 1 if its distance from face with id = 1 is lower than a given threshold or it has id = 2 otherwise, and so on. The i-th face has the label equal to that one of the face having the minimum distance (anyway below the threshold). If there are no close faces in the considered feature space, then a new identifier is introduced. The threshold was set to 0.4 when no information about shot transition detection were used. Two adaptive thresholds were instead introduced to get advantage of the knowledge of the camera shot boundaries. The aforementioned threshold was in fact relaxed to 0.6 when the comparison was made between two face images belonging to the same shot.
At the end, assigned label were compared with true labels manually annotated and the results of this comparison is reported in Figure 7. The advantages in using shot transition detection are evident. Many errors in face matching are avoided by considering different thresholds depending on the availability of shot transition detector information.
About implementation details, the face detector has been implemented in C++. A modified version of the pre-trained CAFFE model of [36] has been used. For facial vector generation, a SE-ResNet-50-256D has been implemented in CAFFE using the pre-trained models built on VGGFace2 dataset as introduced in [5]. The camera shot transition detection has been written in C++ with the OpenCV libraries, while the face indexing illustrated in the last two blocks has been implemented with MATLAB 2019a.

Conclusion
Most relevant works in the literature gather full indexing of videos in real scenarios by exploiting additional media features (e.g. audio and text), but there are application contexts where multimedia data are either not available or reliable. At this aim, a new fully computer vision based video indexing pipeline has been proposed in this work. The proposed pipeline can detect and re-identify people in each image supplying an automatic subject dependent video indexing. Faces are processed without any constraints about appearance, eyeglasses, beard or hairstyle. Moreover, a shot transition detector has been introduced and employed to strengthen the results. The system has been validated and tested in two different typical scenarios where no-multimedia data could be exploited: broadcasted political video documentaries and healthcare therapies sessions about non-verbal skills. Experiments demonstrate encouraging results in the field of on-demand video retrieval, making the system suitable in the two assistive scenarios under  investigation. Future works will consider the possibility to put in place the proposed approach as part of an assistive system to be exploited in healthcare centers for assessing and diagnosing Autism Spectrum Disorders in children.