{'fraction_non_alphanumeric': 0.026775711404884606, 'fraction_numerical': 0.01008290387631638, 'mean_word_length': 4.857283464566929, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bag-of-Genres for Video Genre Retrieval

This paper presents a higher level representation for videos aiming at video genre retrieval. In video genre retrieval, there is a challenge that videos may comprise multiple categories, for instance, news videos may be composed of sports, documentary, and action. Therefore, it is interesting to encode the distribution of such genres in a compact and effective manner. We propose to create a visual dictionary using a genre classifier. Each visual word in the proposed model corresponds to a region in the classification space determined by the classifier's model learned on the training frames. Therefore, the video feature vector contains a summary of the activations of each genre in its contents. We evaluate the bag-of-genres model for video genre retrieval, using the dataset of MediaEval Tagging Task of 2012. Results show that the proposed model increases the quality of the representation being more compact than existing features.


INTRODUCTION
The retrieval of videos by genre is a challenging application, as videos may be composed of visually different excerpts. For instance, a news video can comprise multiple categories, like sports, documentary, health, and others. A video retrieval system aiming at retrieving videos with similar content should be aware of such property in order to obtain better results.
In this paper, we focus on video retrieval by genre based only on visual information. No tags or textual descriptions are considered. One important step in this scenario is feature extraction from videos. There are mainly two kinds of feature descriptors for videos: descriptors that consider motion and descriptors based on isolated frames. Motion-based descriptors usually obtain space-time interest points and extract histograms of those local points or obtain histogram of motion patterns [1]. Descriptors based on isolated frames are usually derived from image feature extraction. Frames are represented individually and then a pooling function can be used to obtain the video feature vector. The advantage of the first kind of descriptors is obviously the encoding of transitions between frames. The advantage of the second kind is the large number of descriptors already proposed for image representation.
Regardless of motion, many of the state-of-the-art solutions for feature extraction are based on visual dictionaries. Such dictionaries are commonly based on local patches, which are semantically poor. Therefore, both kinds of descriptors usually present the same Thanks to CAPES, CNPq, and FAPESP for funding.
property: the video feature vector has few semantics from the human perspective.
In this paper, we present a novel approach for video representation in video genre retrieval tasks, called Bag-of-Genres (BoG). The proposed method is based on dictionaries of genres created from genre classifiers. Each visual word in the BoG model is a genrelabeled region of the classification space defined by the classifier's model. Thus, the final video representation corresponds to an activation vector of its contents to each of the genres in the dictionary. Therefore, each component of the representation model has self-contained semantics and is directly related to a specific concept.
We validated the BoG model in the dataset of MediaEval Tagging Task of 2012. We evaluate the importance of the genre classifier in the model as well as the quality of the BoG representation. Although the genre classifier has low accuracy, the BoG model could work well in the experiments. The results are comparable to the existing baselines, even BoG being much more compact.

RELATED WORK
In this section, we describe related work focusing on works that are based on visual dictionaries and works that aim at including semantics in the representation.
Many solutions exist in the literature aiming at including semantics in the representation. There are techniques in which an image is represented as a scale-invariant response map of a large number of pre-trained generic object detectors [2], which could be seen as a dictionary of objects. Poselets have also been used similarly to a dictionary of poses for recognizing people poses [3]. Labeled local patches have also been used for having a dictionary with more semantics [4]. Boureau et al. [5] also present a way to supervise the dictionary creation. Other approaches can also be considered as related to the intention of having dictionaries with more meaningful visual words [6,7,8] The approach proposed here is closely related to the Bag-of-Scenes (BoS) model [9], in which the video feature vector is an activation vector of scenes. As scenes are more semantically meaningful than local patches, the BoS feature space is semantically richer. Each dimension in the BoS space corresponds to a semantic concept.
The main novelty of BoG in relation to previous works, specially BoS, is that we use a genre classifier as visual dictionary. In the BoS model, the visual dictionary is based directly on the feature vectors of the scenes. The advantages of using a classifier is that it better delineates the frontiers among visual words and tends to be more robust to feature dimensionality. Another advantage is the compact BoG vector, as its dimensionality directly corresponds to the number of genres in the problem.

BAG OF GENRES
In this section, we describe the Bag-of-Genres (BoG) model for video representation. This model is based on a dictionary of genres, in which each visual word corresponds to a decision region of the classification model defined by a genre classifier. Thus, each video is represented by a vector of activations of its frames to each of the genres in the dictionary.
The main advantage of the BoG model is that it relies on elements that have more semantics according to the human perception. Traditional dictionaries based on local features, like SIFT or STIP, are composed by visual words which carry no semantic information, like corners and edges [2]. In the BoG model, as the visual words are genre-labeled regions of the classification space, the activation vector has one dimension for each genre, making it simple to analyze the presence or absence of each genre into a video. Figure 1 shows a flowchart of the stages involved in representing video content using the BoG model. On top, we show how the visual dictionary is created. At the bottom, we show how this codebook is used to represent video content.
The creation of the visual dictionary is performed as follows. Given a set of training videos with known genre labels, we first discard a lot of redundant information, taking only a subset of video frames. Techniques like sampling at fixed-time intervals or summarization methods [10,11] are examples of possibilities for frame selection. In this paper, frames were selected using the well-known FFmpeg tool 1 in a sampling rate of one frame per second. After that, we perform the feature extraction from each of the selected frames in order to encode their visual content into feature vectors. Such features can be any, like for instance, color histograms, GIST, bags of quantized SIFT features. Then, those feature vectors and their associated genre labels are used as input for training a genre classifier. The obtained classification model represents the dictionary of genres used for representing videos.
After creating the visual dictionary, we should represent videos according to the dictionary space. Given an input video, we initially apply frame selection and feature extraction from each frame. After 1 http://www.ffmpeg.org/ (as of January 2015). that, the feature vectors of each frame must be coded according to the dictionary of genres. Each feature vector is classified by the genre classifier, which predicts a genre label for the frame. The labeling process is analogous to the coding step of traditional visual dictionaries [12]. Finally, a normalized frequency histogram is obtained by counting the occurrences of each of the genre labels, forming the bag-of-genres representation for the input video. Such step can be understood as pooling the frame genres [5].
The dimensionality of the bag-of-genres feature space is directly related to the number of genres used for training the genre classifier during the dictionary creation. Therefore, as in many applications the number of genres is small, the bag of genres is usually more compact than existing features.

EXPERIMENTS AND RESULTS
Experiments were conducted on a benchmarking dataset provided by the MediaEval 2012 organizers for the Genre Tagging Task [13]. The dataset is composed of 14,838 videos (3,288 hours) collected from the blip.tv 2 and is divided into a training set of 5,288 videos (36%) and a test set of 9,550 videos (64%). Those videos are distributed among 26 video genre categories assigned by the blip.tv media platform, namely (the numbers in brackets are the total number of videos): art (530), autos and vehicles (21) The main challenge of this collection is the high diversity of genres, as well as the high variety of visual contents within each genre category [14,15].
After frame selection (1 per second), the training set has 3,943,375 frames and the test set has 7,273,996 frames. Different image descriptors were evaluated for extracting features from such frames. The descriptors for encoding color properties are: Auto Color Correlogram (ACC) [16], Color Coherent Vector (CCV) [17], Border/Interior pixel Classification (BIC) [18], and Global Color Histogram (GCH) [19]. The texture descriptors evaluated are: Generic Fourier Descriptor (GFD) [20] and Haar-Wavelet Descriptor (HWD) [21]. For more details regarding those image descriptors, please refer to [22].
The experiments are divided into two phases. The first one evaluates the genre classifier. The second one evaluates the BoG representation for video genre retrieval.

Evaluation of the genre classifier
The evaluation of the genre classifier is important because the quality of the final BoG vector depends on the quality of this classifier. If the genre classifier classifies the frames in wrong genres, the BoG vector will not reflect the correct distribution of video genres. It would be similar to have a bad coding step in traditional visual dictionaries of quantized local features: wrong visual word labels would be assigned to image patches, resulting in a bad bag of visual words. Therefore, the BoG model depends on a good genre classifier.
To create the visual dictionary, we trained a linear SVM (c = 1.0) using features extracted from the training videos. The genre (label) of each training frame is the same of the video from where it was extracted. The training of the genre classifier was based on randomly selecting the same number N of frames per genre. We varied N in 100, 500, and 800 frames per genre. The remaining frames were used for testing. It is worth mentioning the amount of frames used in this evaluation: almost 4 million of the training videos and more than 7.2 million of the test videos (no frames of the test videos were used for training the genre classifier). For running SVM, we used the LIBSVM package 3 [23]. Figure 2 presents the classification accuracy for the evaluated descriptors. Notice that the classification accuracies are low for all the descriptors, creating a very challenging scenario for the BoG model, as we explained previously. The best results were obtained for the SVM model learned on 800 training frames per class. This model was used for representing the test videos using the BoG approach in the following experiments.

Evaluation of the BoG representation
The following experiments evaluate the BoG model for video genre retrieval. Each video in the test set was represented by a bag of genres using the genre classifiers learned on the training step. With the BoG of each video, a given test video was used as query for the rest of videos in the test set, which were ranked according to the Euclidean (L2) distance between their BoGs. For each genre, around five percent of the test videos were randomly selected and used as queries. Five replications were performed in order to ensure statistically sound results. Presented results refer to the average scores and their respective 99% confidence intervals, which were computed based on the mean and standard deviation of each replication.
We compared the BoG approach against with two baselines: Histogram of Motion Patterns (HMP) [1] and Bag of Scenes (BoS) [9]. To make a fair comparison, these approaches were configured with their best settings based on the results reported in [24]. The distance function used for feature comparison is the Euclidean (L2) distance.  Fig. 2. Evaluation of the genre classifier. All descriptors generated low discriminating genre classifiers (accuracy below 50%), creating a challenging scenario for the BoG model.
The retrieval effectiveness was assessed using the precision at the top 10 retrieved items (P10) and Mean Average Precision (MAP).
In Figure 3, we compare the BoG representations and the baseline methods with respect to the MAP and P10 measures. As we can observe, the performance of the BoG representations are better considering the MAP measure. MAP is a good indication of the effectiveness considering all positions of obtained ranked lists. P10, in turn, focuses on the effectiveness of the methods considering only the first positions of the ranked lists.
The BoG approach achieved the best scores using BIC as the frame descriptor (used as basis for the genre classifier). Notice that BoGBIC performs better than the baseline methods for MAP, however the same does not happen for P10. BIC was the best descriptor for the genre classifier in the test set (see Section 4.1), making it also better for generating the BoG vector.
We also performed paired t-tests to verify the statistical significance of the results. For that, the confidence intervals for the differences between paired averages of each class were computed to compare every pair of approaches. If the confidence interval includes zero, the difference is not significant at that confidence level. If the confidence interval does not include zero, then the sign of the difference indicates which alternative is better. Table 1 presents the 99% confidence intervals of the differences between BoGBIC (the best configuration of BoG) and the baseline methods for the MAP and P10 measures, respectively. Notice that the confidence intervals for BoGBIC and BoS include zero and, hence, the differences between those approaches are not significant at that confidence level. On the other hand, the performance of BoGBIC and HMP are not statistically different for MAP, whereas BoGBIC performs worse than HMP for P10. This method is based on motion information and, hence, it does not consider visual properties of video frames in an independent manner. Figure 4 compares the individual scores obtained for each class in terms of MAP and P10 measures. It is interesting to note the differences in responsiveness of the different approaches with respect to each of the genres. For MAP, BoGBIC performs better than the baseline methods for most of the classes (13 of 26). For P10, BoGBIC provides a good discriminative power on genres like "school and education" and "web development and sites".
The key advantage of the BoG model is its computational effi-    ciency in terms of space occupation and similarity computation time.
In our experiments, the BoG vector corresponds to a 26-bin histogram, which represents a reduction of 74% in relation to the BoS vector (100-bin histogram) and is two orders of magnitude smaller than the HMP vector (6075-bin histogram), making our approach more suitable for real-time processing. Although the effectiveness the BoG approach is not superior to the baseline methods, the obtained results show the potential of the idea. As we explained previously, the quality of the genre classifier is important for the BoG quality. Our genre classifiers obtained less than 50% of accuracy in the training set and less than 30% in the test set, probably limiting the quality of the BoG representation. Another limitation is the dataset used. As all the frames of a video have the same label, visually different frames may be of the same genre, harming the classifier.

CONCLUSIONS
In this paper, we presented a new video representation for video genre retrieval, named Bag-of-Genres. This representation model relies on a dictionary of genres, which is created from a genre classification model learned on the training frames. Different from traditional dictionaries based on local features (e.g., SIFT or STIP), here, visual words correspond genre-labeled regions of the classification space. Therefore, each dimension of the feature space spanned by such a model is associated to a semantic concept.
Our approach was validated in the dataset of MediaEval Tagging Task of 2012. Our experiments evaluated the importance of the genre classifier in the model as well as the quality of the BoG representation. In these experiments, the BoG model has performed well despite the low accuracy of the genre classifier. The results demonstrated that our approach performs similar to state-of-the-art methods, but using a much more compact representation.
We can think about ways of improving the BoG model. For instance, a smarter strategy for feature extraction and classification may enable to create more informative visual dictionaries and, hence, improve the video representation.
Future work includes the evaluation of other methods for feature extraction, as well as perform an extensive study on classification strategies to be used in the creation of visual dictionaries. We also would like to evaluate the use of a dataset of scene images to create