{'fraction_non_alphanumeric': 0.029960584742802974, 'fraction_numerical': 0.01701342114453924, 'mean_word_length': 4.832533100538338, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Strategies for Searching Video Content with Text Queries or Video Examples

The large number of user-generated videos uploaded on to the Internet everyday has led to many commercial video search engines, which mainly rely on text metadata for search. However, metadata is often lacking for user-generated videos, thus these videos are unsearchable by current search engines. Therefore, content-based video retrieval (CBVR) tackles this metadata-scarcity problem by directly analyzing the visual and audio streams of each video. CBVR encompasses multiple research topics, including low-level feature design, feature fusion, semantic detector training and video search/reranking. We present novel strategies in these topics to enhance CBVR in both accuracy and speed under different query inputs, including pure textual queries and query by video examples. Our proposed strategies have been incorporated into our submission for the TRECVID 2014 Multimedia Event Detection evaluation, where our system outperformed other submissions in both text queries and video example queries, thus demonstrating the effectiveness of our proposed approaches.


Introduction
As we see an unprecedented growth of user-generated videos on the Internet, it is crucial to have an effective indexing and searching mechanism for these videos. To perform search, current existing video search engines mainly rely on user-generated text metadata. However, text metadata is often not a comprehensive representation of the video as: 1) users often do not provide metadata, and 2) even if users do provide metadata, a user cannot possibly annotate all facets of the video. Therefore, content-based video retrieval gap between a pure text-based query and the non-semantic representation of a video. The final challenge is the indexing/search challenge. As new features are used to represent the visual and audio channels, traditional text-search techniques are not directly applicable, and new search techniques need to be developed. To enable search over large video collections, these new search techniques have to be both effective and efficient.
In light of the aforementioned challenges, we propose multiple strategies to tackle these problems. For the low-level feature extraction challenge, we propose two different features to significantly enhance CBVR performance. The first feature is a variant of the Improved Dense Trajectory feature 3)4) (Section 4. 1), and the second feature is a deep learning feature (Section 4. 2) trained on ImageNet 5)6) data. For the text/video semantic gap challenge, we propose a method which utilizes large amounts of weakly-labeled videos to learn semantic concept detectors encompassing a large vocabulary (Section 5). This enlarged vocabulary is crucial in bridging the semantic gap between a text-query and non-semantic video representations. For the indexing/search challenge, we first propose to utilize Explicit Feature Maps 7) and Product Quantization 8) to perform efficient yet effective video search (Section 6. 1). We then propose a novel fusion method called Multistage Hybrid Late Fusion (MHLF) to effectively fuse search results from multiple feature modalities (Section 6. 2). Finally, we propose a self-paced reranking method 9) to automatically enhance search results through pseudo-relevance feedback (Section 6. 3). The aforementioned methods were all integrated into our TRECVID MED 2014 system, which was the leading system in all eight MED subtasks, thus demonstrating the effectiveness of our proposed strategies.
In the following sections, we first give an overview of a general CBVR system and related work in Section 2. Then we summarize our results in the TRECVID MED 2014 task in Section 3. Details of each proposed strategy are given in Sections 4, 5 and 6. Finally, Section 7 concludes the paper.

Content-based Video Retrieval Preliminaries
A general pipeline of a CBVR system is shown in Figure 1. There are mainly two phases: the offline phase and the online phase. In the offline phase, low-level and semantic features are extracted for a large video repository and indexed so that the online phase is sufficiently efficient. The semantic features are predictions of semantic concept detectors, which takes low-level features as input and predicts whether a given concept such as dog, cat, or car exists in a video. In the online phase, users will provide different types of queries to search for relevant videos. There are mainly two types:  The model is trained based on the features extracted in the offline phase. Then, the model is applied to the video repository to search for other related videos. Lastly, the search results goes through an iterative reranking process, which performs pseudo-relevance feedback to automatically improve the search results. For searching by text queries, the first step is semantic query generation, where the text query is mapped to the system vocabulary. The system vocabulary constitutes of all concepts that could be detected by the available semantic concept detectors. Then the generated semantic query is utilized to perform semantic search. The initial ranked list also goes through the reranking process to acquire a more accurate ranked list. In the following sections, we will explain the details and also briefly review the related work for each component.

1 Searching by Video Examples
For the query by video examples scenario, low-level features combined with discriminatively learned search models play the key role in achieving good performance 10)11) . In the following sections, we will review the related work on these two topics.
( 1 ) Low-level Features Many low-level audio and visual features have been utilized to enhance CBVR performance. The most popular low-level audio feature used is the Mel-Frequency Cepstral Coefficients (MFCC) 11) , which have been shown to be the most cost-effective feature 12) . Other audio features including Acoustic Unit Descriptors 13) and Large-scale Pooling Features 14) have also been utilized.
Low-level visual features can be split into two categories: static image features and motion features. Static image features are essentially image-based features extracted from all or selected frames of a video. The temporal relation between the frames are not taken into account. Before the introduction and success of deep features, mainstream static image features were frequently hand-crafted SIFT-based features 15)16) . Currently, deep convolutional neural network (DCNN) features 11)17) are significantly outperforming hand-crafted features and represent the current mainstream. In this paper, we present architectural improvements for static-image deepnetworks to enhance CBVR performance.
Motion features utilize the temporal relations between frames to capture motion characteristics of a video. Optical flow is typically used to compute motion features. Currently, one of the most effective features is Improved Dense Trajectories (IDT) 18) , which significantly outperforms the previously proposed popular motion features such as Space Time Interest Points (STIP) 19) and Motion SIFT (MoSIFT) 20) . In this paper, we present two enhancements which further improves the performance of IDT.
One problem with the previously mentioned low-level features is that they will generate a different number of feature vectors depending on the length, resolution and contents of the video, thus leading to varying length vector representations for each video. It is very difficult to compare two videos with different length representations. Therefore, the varying length representations of each video need to be converted to a fixed-length vector representation, thus many different encoding/pooling techniques have been proposed, including Bag-of-Words (BoW) 21) and Spatial Pyramid BoW (SpBoW) 22) , Fisher Vectors (FV) 23) 24) , and Vector of Locally Aggregated Descriptors (VLAD) 25) . FV and VLAD are the current mainstream encoding methods 17) .
Overall, a "complete" feature is a combination of a lowlevel feature and an encoding method. For example, SIFT can be encoded with SIFT-SpBoW or SIFT-FV, and MFCC can also be encoded with BoW (MFCC-BoW) or FV (MFCC-FV) respectively. Once these encodings have been computed, they are indexed for the subsequent learning-based search.
( 2 ) Learning-based Search There are two key components to learning-based search: the learning component, and the fusion component.
Utilizing machine learning models such as Support Vector Machines (SVM) 10)26) and Kernel Ridge Regression (KRR) 11) has shown to be very effective for querying with video examples. The main idea is to treat the example videos as positive training data, and when combined with a large pool of negative videos, a classifier can be trained to determine whether an input testing video is relevant or not. Fusion enables the incorporation of search results from different features which capture the multiple aspects of a video. The main challenge of fusion is to effectively estimate the reliability of each feature source so that the fusion algorithm knows which features to rely more on when dealing with different videos. Many fusion method such as early fusion, late fusion, double fusion 27) and other more complex methods 28) have been proposed. In this paper, we propose a Multistage Hybrid Late Fusion method, which shows superior performance and robustness over other fusion methods.
( 3 ) Reranking Reranking utilizes pseudo-relevance feedback (PRF) to automatically enhance an initial rank list. The intuition of PRF is that the top-ranked results in an initial rank list are highly likely to be correct, and adding these instances back into the training set may improve performance. This simple method has shown to be effective in many different scenarios. However, previous PRF methods usually operate on a single ranked list 29) , but the CBVR task inherently outputs multiple ranked lists from different features, and effectively fusing these ranked lists becomes a challenging task 30) .
In this paper, we introduce self-paced reranking, which further improves the performance of existing PRF approaches. Our system incorporates MMPRF 30) and SPaR 9) to conduct reranking, in which MMPRF is used to assign the starting values, and SPaR is used as the core reranking algorithm. The reranking is inspired by the self-paced learning proposed by Jiang et al. 9) , in that the model is trained iteratively as opposed to simultaneously. Our methods are able to leverage high-level and low-level features which generally leads to increased performance 31) . The high-level features used are ASR, OCR, and semantic visual concepts. The low-level features include DCNN, IDT and MFCC features.

2 Searching with Text Queries
This scenario takes a pure-text query as input, and outputs a ranked list of relevant videos. It is an interesting task because it resembles a real-world video search scenario, where users typically search videos by using query words instead of providing example videos.
The main challenge of the text-to-video search scenario is to bridge the semantic gap between text and video. In current state-of-the-art systems, this gap is usually bridged with automatic speech recognition (ASR), optical character recognition (OCR), and semantic concept detectors. Semantic concept detectors are trained to detect whether a certain object, scene, or action exists in a video or not. Given a pool of concept detectors, these detectors can be applied on an input video to acquire a semantic feature representation of the video, which corresponds to the confidence score of detecting a concept in the video. This feature representation is very different from the low-level feature representations, where each dimension in the vector does not have a clear semantic meaning. Popular datasets to train concept detectors include the ImageNET 5) dataset, the SUN397 32) scene dataset, and the TRECVID Semantic Indexing (SIN) 1) dataset. To train effective static-image-based detectors, the current mainstream is deep convolutional neural network models 6)33) . To train videobased detectors, the current mainstream is combining deep static-image detectors with motion features such as Improved Dense Trajectories 18) .
According to Jiang et al. 2)30) , a text-to-video search system consists of three major components, namely Semantic Query Generation (SQG), Semantic Search and Reranking/PRF as shown in Figure 1. The Semantic Query Generation component translates the description of the user's information need into a set of multimodal system queries that can be processed by the system. There are two challenges in this step. Since the semantic vocabulary of the system is usually limited, the first challenge is to map the user's query words into the system vocabulary. The second challenge is assigning a given query word its modality as well as its weight associated with that modality. A preliminary study of these challenges is detailed in Jiang et al. 2) .
The semantic search component retrieves multiple ranked lists for a given text query. Our system incorporates various retrieval methods such as the Vector Space Model, tf-idf, BM25, language model 34) , etc. Surprisingly, a better retrieval model on worse features actually outperforms a worse retrieval model on better features. This observation suggests that the role of retrieval models in our semantic search system may be underestimated in much current research. After retrieving the ranked lists for all modalities, we apply a normalized fusion to fuse different ranked lists according to the weights specified in SQG.
Reranking is also performed for text query search. One key advantage of reranking is that it "bridges" the semantic search and the learning-based search 30) . Once the text query search component generates an initial ranked list, the positives in this ranked list can be used to perform learning-based search, which can often further improve search performance.

TRECVID Multimedia Event Detection 2014
The TREC Video Multimedia Event Detection (MED) 1) task is a standardized task held every year since 2010 to evaluate the performance of different CBVR systems on the MED task. Different CBVR systems are presented with multiple queries, and the CBVR system needs to retrieve relevant videos from the evaluation set. Example queries are shown in Figure 2. The query consists of two parts, the textual query and the video examples. Different parts of the query will be utilized for different query settings as described below. In addition to what was given for 000Ex, 100 positive videos were given. The background set, which can be viewed as the negative videos, contained 4992 videos. The validation set, which is also know as MEDTEST14, contained around 24,000 videos. The testing set contained around 198,000 videos (8000 hours of video) which does not contain any text metadata. In the competition, competitors trained their system on the positive and background sets, and then tuned the system on the validation set. Finally, the resulting system performed search over the testing set and the results were submitted to the organizers. Label information was only available for the positive, background and validation sets. The labels for the testing set were never released to prevent overfitting on the testing set. In the competition, there were two types of queries, pre-specified and ad-hoc. For pre-specified queries, the names of events were given a few months beforehand, so participants could design specialized detectors for these events. On the other hand, ad-hoc queries were given a few days before the deadline, leaving no time to design specialized detectors. There were a total of 20 events/queries for the MED14 pre-specified run and 10 events for the MED14 ad-hoc run. The evaluation metric used was Mean Average Precision (MAP) 1) .

CMU MED14 Submission Overview
For MED14, we had a system 11) for text queries (SQ, 000Ex) and another system for query by video examples (010Ex, 100Ex). For text queries, our system utilized ASR,     11) . Figures 3 and 4 present the results of our system and other competing systems on the MED14 task. We can see that our system is significantly better than other competing systems, thus demonstrating the effectiveness of our strategies. In the following sections, we will detail each of our strategies. However, as the labels for the testing set were never released, we can only present experimental results on the 20 pre-specified events on the validation set MEDTEST14.

1 Enhancements for Improved Dense Trajectories
We improve the original Improved Dense Trajectory 18) in two ways. First, temporal scale-invariance is achieved by extracting features under different video playback speeds, which are generated by skipping frames at certain intervals. We denote this new way of feature extraction as Multi-skIp Feature Stacking (MIFS) 4) . Different from what has been described in Lan et al. 4) , we use the combination of level 0, 2 and 5 to balance speed and performance.
Second, we propose a new space-time encoding method, dubbed Space-Time Extended Descriptors (STED), that attaches spatial (x, y) and temporal (t) location information to the raw features after PCA-projection 4) .
As illustrated in Table 1, by using MIFS, we improve MAP of both 100Ex and 010Ex on MEDTEST14 by about 2%, absolute. We further add STED to the results of MIFS and compared it with Spatial Pyramid Matching (SPM) 22) , a classical space-time encoding method. As can be seen, STED can get similar or better results compared to the results of only using MIFS. SPM can also improve the baseline results, but due to its high dimensionality, it needs large space for storing Paper ( 5 ) 5 the resulting feature vectors and is computationally expensive to run the classifiers, thus STED is a more space efficient alternative to incorporate spatial and temporal information into a feature. For details, please see Lan et al. 4) .

2 Features from ImageNet DCNN Models
In order to leverage the powerful deep learning models in MED, we improved existing DCNN models in two directions: 1) by utilizing more data and 2) by modifying the network structure. In total, we have extracted a total of 15 different Deep Convolutional Neural Network (DCNN) features in our MED14 system. All models were trained on different subsets of ImageNet.
We utilized more data by training 3 models 12) on the whole ImageNet dataset consisting of around 14 million labeled images and 28,000 classes. We took the networks at epoch 5, 6 and 7 and generated features for MED keyframes using the first fully connected layer and probability layer. To generate video features from keyframe-level features, we used both maximum pooling and average pooling for the probability layer and only average pooling for the fully connected layer. This procedure results in 9 DCNN-ImageNet representations for each video.
To explore the performance of deep models under varying network structures, another 5 models were trained on the standard ILSVRC 2012 dataset 5) which had around 1.28 million images belonging to 1,000 classes. Two models were trained with six convolutional layers, two models were trained with smaller filters, and one was trained with a larger number of filters. A multi-view representation was used for one of the models. The network structure is as described in Zeiler et al. 37) . Except for different structures among models, the models with the same structures differ in initialization. The training process was tuned on the ImageNet ILSVRC 2012 validation set with 50 thousand images. These models result in another 6 different feature representations. More details and also some further improvements after the MED14 evaluation are described in 17) . Figure 5 illustrates the improvements on different network structures we have explored within the ILSVRC 2012 training setting. We started with the standard AlexNet but with 6 convolutional layers and the features were computed from the 1,000 dimensional probability layer. Intuitively, the probability output provides a semantic feature representation for each video, where each dimension corresponds to a specific object. We can regard this feature as Bag-of-Words with a vocabulary of 1,000 visual objects. The features were then fed into a χ 2 -exponential SVM for classification. We only achieved 0.246 MAP on MEDTEST14 100Ex, which is far below IDT. We then explored features from other layers, e.g., pool 5 , fc 6 and fc 7 . Adding multiple layers into the video representation increases the MAP to 0.277.
We further explored a wider network by doubling the number of filters in each convolutional layer. For example, the standard AlexNet had 256 filters in the 5-th convolutional layer, while we explored the 5-th convolutional layer with 512 filters. This way, the network learns more complex patterns in the images and improved the MAP to 0.288. Following Zeiler et al. 37) , we made the filter size of the first convolutional layer smaller, i.e., reducing it from 11 to 4, and decreased the stride of this layer from 4 to 2. Though this dramatically increased training time for the network due to much more time-consuming convolutional operations on the first convolutional layer, the smaller filter size and stride helped the network capture finer patterns and boosted performance to 0.325, which outperforms the previous versions significantly. In the stages described above, we only utilized a single crop from the central 224-by-224 pixels of the video frames, which may lose some helpful visual information. Therefore, we generated a multi-view DCNN feature by producing 10 crops per input frame, which included the top-left, top-right, bottom-left, bottom-right and center crops along with their corresponding mirrored crops. The features obtained from the 10 views are subsequently averaged together to acquire a single vector representation. This further improved performance to 0.332. The whole exploration of utilizing features extracted from ImageNet pretrained models with different structures raised performance from 0.246 to 0.332, which is a big improvement over state-of-the-art hand-crafted features.

Bridging the Text/Video Semantic Gap
Our shot-based semantic concepts were directly trained on video shots and not still images for the following two reasons: 1) shot-based concepts have minimal domain difference; 2) this allows for action detection. We have found that detectors trained on still images usually do not work well on video, which may suggest that the domain difference between static images and video data such as MED data is significant.
The shot-based semantic concept detectors were trained with our pipeline based on our previous study on Cas-cadeSVM and a new study on self-paced learning 38) 39) . Our system included more than 3,000 shot-based concept detectors which were trained on around 2.7 million shots using the standard improved dense trajectory features 18) . The detectors are generic and include people, scenes, activities, sports, and fine-grained actions described in 40) . The detectors were trained on several datasets including Semantic Indexing 1) , YFCC100M 41) and Google Sports 42) . YFCC100M and Google Sports are weakly-labeled datasets, i.e. the labels for each video were inferred from the metadata of the videos and not annotated by a human. The notable increase in quantity and quality of our detectors significantly contributed to the improvement in the text-to-video system performance.
Training large-scale concept detectors on big data is very challenging, thus requiring research on both theoretical and practical perspectives. Regarding theoretical progress, we adapted self-paced learning theory, which provided theoretical justification for concept training. Self-paced learning is inspired by the learning process of humans and animals 39)43) , where samples were not learned randomly but organized in a meaningful order: from easier samples to gradually more complex ones. We advanced the theory in two directions: augmenting the learning schemes 9) and learning from easy and diverse samples 38) . The two studies offer a theoretical foundation for our detector training system.
As for practical progress, we optimized our pipeline for high-dimensional features (around 100K dimensional dense vector). Specifically, we utilize large shared-memory machines to store the kernel matrices, e.g. 512GB in size, in memory to achieve 8 times speedup in training. This enabled us to efficiently train more than 3,000 concept detectors over 2.7 million shots by self-paced learning 38) . We use around 768 cores in Pittsburgh Supercomputing Center for about 5 weeks, which could be roughly broken down into two parts: low-level feature extraction for 3 weeks and concept training for 2 weeks. For testing, we converted our models to linear models to achieve around 1,000 times speedup in prediction.
In summary, our theoretical and practical progress provided the foundation for developing critical tools for largescale concepts training on big data. For instance, if we had 500 concepts over 0.5 million shots, then, optimistically speaking, we can finish training within 48 hours on 512 cores, including the raw feature extraction. After getting the models, the prediction for a shot/video only takes 0.125s on a single core with 16GB memory.

1 Efficient Learning-based Search
The most natural way for a human to utilize a system is through an interactive process. Therefore, to strive for interactive MED, we targeted completing learning-based search over 200,000 videos in 15 minutes on a single machine. This is a big challenge for the query by video example pipeline, as we utilized 47 features and around 100 classifiers (SVM & KRR) to create the final ranked list. The text search pipeline is a lot simpler thus timing is not a big issue. Therefore, we will focus on the query by video example system in the remaining section. To speed up, we performed optimizations in three different directions: 1) decreasing computational requirements, 2) decreasing I/O requirements and 3) utilizing GPUs. Computational requirements were decreased by replacing kernel classifiers with linear classifiers. I/O requirements were decreased by compressing features vectors with Product Quantization 8) (PQ). GPUs were utilized for fast linear regression and prediction.
( 1 ) Replacing Kernel Classifiers by Linear Classifiers Kernel classifiers are slow during prediction time because to perform prediction on a testing video vector, it is often required to compute the dot-product between the testing video feature and each vector in the training set. For MED14, we had around 5000 training videos, so 5000 dot products were required to predict one video. This is too slow, as preliminary experiments showed that prediction of improved trajectory fisher vectors (IDT-FV, 109056 dimensions) on 200,000 videos required 50 minutes on a NVIDIA K-20 GPU. Therefore, to accelerate this process, we switched to linear classifiers, which requires only one dot product per testing vector, thus in theory we have sped up by 5000x. However, bag-of-word features do not perform well with linear kernels. Therefore, we used the Explicit Feature Map (EFM) 7) to map all bag-of-word features to a linearly separable space before applying the linear classifier. As the EFM is an approximation, we run the risk of a slight drop in performance. Figures 6 and 7 show the performance difference before and after EFM approximations. For most features, we suffer a slight drop in performance, which is still cost-effective given that prediction speed was sped up by 5000x.
( 2 ) Feature Compression with Quantization In order to improve I/O performance, we compressed our features using Product Quantization 8) (PQ). Compression is crucial because reading uncompressed features can take a lot of time. PQ compresses feature vectors by first splitting each feature vector into multiple chunks, and then quantizing each chunk with a 256 word codebook. A 256 word codebook Paper is ideal because cluster assignments can be stored with 1 byte. Therefore, the chunk, which we set to 8 floating point numbers (32 bytes) in our system, is simply represented by 1 byte, thus achieving 32X compression. Also, faster classifier prediction can be done based on the PQ codebooks. More details are in Jegou et al. 8) and Yu et al. 44) . However, as PQ performs lossy compression, the quality of the final ranked list may degrade. Figures 6 and 7 shows the performance drop before and after PQ approximation. We can see that there is nearly no performance drop before and after PQ. Figure 8 further shows MEDTEST14 010Ex performance when performing quantization under different compression ratios. We show performance of two quantization methods, PQ and Uniform Quantization (UQ). The basic idea of UQ is to quantize each dimension of all feature vectors into k bins, and each dimension can be represented with log 2 (k) bits. As we can see, PQ and UQ have similar performance. The problem with UQ is that one can at most achieve 32X compression when k = 2, but PQ can achieve higher compression ratios by adjusting the size of each chunk.
( 3 ) Utilizing GPUs for Fast Linear Regression * and Linear Classifier Prediction Following the TRECVID MED 2014 guidelines, we were limited to a single workstation for learning-based search. Therefore, we utilized all available computing resources on our workstation, which includes CPUs and GPUs. Exploiting the fact that matrix inversion on GPUs are faster than CPUs, we trained our linear regression models on GPUs, which is 4 times faster than running on a 12 core CPU. We also ported the linear classifier prediction step to the GPU, which runs as fast as a 12 core CPU. Our workstation had 2 Intel(R) Xeon(R) CPU E5-2640 6 core processors, 4 NVIDIA TESLA K20's, 128GB RAM, and 10 1T SSDs setup in RAID 10 to increase I/O bandwidth.
( 4 ) Overall Speed Improvements As both EFM and PQ are approximations, we quantified the drop in performance when both methods were used. The results are shown in Table 2. We see a 3% relative drop in performance for 100Ex and a slight gain in performance for 010Ex. Despite slight drop in performance, speed has been substantially decreased. We have sped up our system by 16 times for learning-based search with a cost of 3% relative drop in performance, which is negligible given the large efficiency gain. * For linear features, the KRR model effectively becomes linear regression.
α : Used in our MED13 system 27) . β : Extrapolated timing for our MED13 system 27) . γ : A modified MHLF was used so that it is compatible with features of the MED13 system, thus leading to slightly different numbers than Table 3.

2 Multistage Hybrid Late Fusion Method
For a given query, the goal of fusion is to learn the weights of different modalities according to the effectiveness of each feature. A simple way to learn modality specific weights can be by training a linear regression model on held-out data from the training set. However, this approach is usually not as stable as if the held-out set is small, the learned weights tend to overfit. To this end, we propose a new learning based late fusion algorithm, named the "Multistage Hybrid Late Fusion" (MHLF) as shown in Figure 9. The MHLF is designed based on the following three key observations: 1. Ranking information is not explicitly modeled in the prediction scores. Therefore, step 1 in Figure 9 augments the original prediction scores with ranking information. 2. Prediction scores from different features contain duplicate information and should not be naïvely averaged.
Duplicate information comes from different features using the same basic feature. For example, SIFT-BoW and CSIFT-FV are all SIFT-based and their ranked lists are usually highly correlated. We propose to model such highly correlated ranked-lists as a generative process. The assumption is that there are many "essential features", whose classifiers generate noise free ranked lists. However, these essential features goes through a duplication and noisy perturbation process, thus what we observe are noisy prediction results. Therefore, to recover the essential features, we perform PCA-Tree clustering as shown in step 2 of Figure 9. The cluster centers corresponds to a "cleaner" version of the prediction results and can be viewed as an estimate of an essential feature. These recovered essential features, and also the original prediction scores are all provided to the next hybrid fusion step. 3. Prediction scores contain random noise and directly learning fusion weights on top may lead to overfitting. To deal with this issue, MHLF utilizes hybrid strategies to acquire a more robust fusion weight estimate. The intuition is that each fusion strategy can be viewed as a random observation of a "ground-truth fusion strategy". Since there is no single fusion strategy that performs better than others on all queries, sampling multiple strategies and averaging them is a simple and effective method to acquire a more stable estimate of fusion weights. The key fusion strategies include: ( 1 ) Average fusion: each feature gets equal weight.
( 2 ) Single-AP: the weights of each feature is its average precision (AP) on the held-out set. ( 3 ) Leave-One-Out: the weights of a feature is the AP performance drop when removing this feature from an average fusion run. ( 4 ) SGD-AP: performs stochastic gradient descent which maximizes average precision as the loss function.
Results on the key features of MEDTEST14 and final fusion results are shown in Table 3. All these results were based on prediction scores from 32X PQ. As we can see, MHLF is superior than average fusion and linear regression fusion. We also performed robustness tests on our fusion algorithm as shown in Figure 10. In this experiment, we randomly removed a subset of features from the original set of 47 features and ran the different fusion methods. As we can see, as we gradually remove features, MHLF is still consistently better than the other two baseline fusion algorithms, thus demonstrating the robustness of MHLF.
From Table 3, we can also compare the relative performance of single features and encoding methods. For static image features, it is clear that multi-view DCNN outperforms handcrafted features such as SIFT and CSIFT. For motion features, MIFS + STED is significantly better than MoSIFT and STIP. Encoding wise, we can see that Fisher vector (FV) encodings are in general better than BoW encodings. Also, in general KRR performs better than SVM in the 010Ex scenario, so for the 010Ex fusion runs we only used pre- diction results from KRR. Finally, if there were resource constraints in feature extraction, combining the 3 core features: MFCC BoW, MIFS + STED, and multi-view DCNN can achieve around 90% of the full system's performance. Among these 3 core features, MFCC excels on events such as "Tuning musical instrument" and "Town hall meeting", where audio such as instrument sounds or speech is an important cue. MIFS + STED performs well on events such as "Rock climbing" and "Winning a race without a vehicle", where the action of people is crucial in determining if a video is relevant. Finally, multi-view DCNN achieves high performance on events which have discriminative objects, such as honeycomb for the "Beekeeping" event, and cars in the "Parking a vehicle" event.
6. 3 Self-Paced Reranking Our PRF system was implemented according to Self-Paced Reranking (SPaR) detailed in Jiang et al. 9) . SPaR represents a general method of addressing multimodal pseudo relevance feedback for SQ/000Ex video search. As opposed to utilizing all samples to learn a model simultaneously, the proposed model is learned gradually from easy to more complex samples. In the context of the reranking problem, the easy samples are the top-ranked videos that have smaller loss. As the name "self-paced" suggests, in every iteration, SPaR examines the "easiness" of each sample based on what it has already learned, and adaptively determines their weights to be used in the subsequent iterations. The mixture weighting/scheme self-paced function was used, since we empiri-cally found it outperforms the binary self-paced function on the validation set 2) . Since the starting values can significantly affect final performance, we used the reasonable starting values generated by MMPRF 30) . The high-level features used were ASR, OCR, and semantic visual concepts. The lowlevel features were DCNN, IDT and MFCC features. We did not run PRF for SQ since our 000Ex and SQ runs are very similar. The final results were computed by averaging the initial ranked list with the reranked list. This is beneficial because for the 000Ex case, the initial ranked list is from semantic search (high-level features), whereas the reranked list is from learning-based search (low-level features), and leveraging high-level and low-level features usually yields better performance 31) . To be prudent, the number of iterations is no more than 2 in our final submissions.
The contribution of our reranking methods is evident because the reranking method is the only difference between our noPRF runs and PRF runs as shown in Figure 3 and 4. According to the MAP on the testing set of MED14, our reranking method boosted the MAP of the 000Ex system by a relative 16.8% for pre-specified events and a relative 51.2% for ad-hoc events. Besides, it also boosted the 010Ex system by a relative 4.2% for pre-specified events, and a relative 13.7% for ad-hoc events. This observation is consistent with the ones reported in previous work 9) 30) . Note that the ad-hoc queries are very challenging because the query is unknown to the system beforehand. As we can see, our reranking methods still managed to yield significant improvement on ad-hoc events. More reranking results on MEDTEST14 data can be found in Jiang et al. 2) .
It is interesting that our 000Ex system for ad-hoc events outperforms 010Ex systems from many other teams. In MED14, the difference between the best 000Ex with PRF (17.7%) and the best 010Ex noPRF (18.2%) is marginal. In MED13, however, this difference was very large where the best 000Ex and 010Ex system was 10.1% and 21.2% * respectively. This observation suggests that the gap of real-world 000Ex event search system is shrinking rapidly. We attribute the improvement of the 000Ex system to the following key reasons: 1) improved semantic concept detectors (Section 5), 2) improvement achieved by the reranking algorithm SPaR, and 3) reasonable queries formulated by human experts.

Conclusion and Future Work
We have described multiple strategies to enhance both the accuracy and speed of content-based video retrieval systems. Overall, the main conclusions are: 1) IDT-based and CNN-based features are the current best motion and static image feature, 2) semantic concept detectors trained from big data are effective, 3) EFM and PQ compression can significantly speed up the system with only a negligible drop in accuracy, 4) MHLF fusion, which fuses multiple fusion strategies, is robust, and 5) reranking is an effective way to enhance accuracy. Looking into the future, we believe that current systems can already achieve reasonable accuracy, but speed is still a big issue. Efficiently extracting features, indexing and searching the billions of videos online will be the next big challenge.  He is a Principal Systems" Scientist in the CMU Computer Science Department and also a faculty member with CMU's Language Technologies Institute. His research combines the areas of multimedia analysis and retrieval, man-machine interfaces, language processing, and machine learning. He is currently leading the Informedia project which engages in understanding of video data ranging from news to surveillance, Internet video for applications in general retrieval as well as healthcare.