{'fraction_non_alphanumeric': 0.0159563724500101, 'fraction_numerical': 0.0020197939810139365, 'mean_word_length': 5.068627450980392, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Language-driven video retrieval

We present an approach to searching large video corpora for video clips which depict a natural-language query in the form of a sentence. This approach uses compositional semantics to encode subtle meaning that is lost in other systems, such as the difference between two sentences which have identical words but entirely different meaning: The person rode the horse vs. The horse rode the person. Given a videosentence pair and a natural-language parser, along with a grammar that describes the space of sentential queries, we produce a score which indicates how well the video depicts the sentence. We produce such a score for each video clip in a corpus and return a ranked list of clips. Furthermore, this approach addresses two fundamental problems simultaneously: detecting and tracking objects, and recognizing whether those tracks depict the query. Because both tracking and object detection are unreliable, this uses knowledge about the intended sentential query to focus the tracker on the relevant participants and ensures that the resulting tracks are described by the sentential query. While earlier work was limited to single-word queries which correspond to either verbs or nouns, we show how one can search for complex queries which contain multiple phrases, such as prepositional phrases, and modifiers, such as adverbs. Video search engines lag behind text search engines in their wide use and performance. This is in part because the most attractive interface for finding videos remains a naturallanguage query in the form of a sentence but determining if a sentence describes a video remains a difficult task. This task is difficult for a number of different reasons: unreliable object detectors which are required to determine if nouns occur, unreliable event recognizers which are required to determine if verbs occur, the need to recognize other parts of speech such as adverbs or adjectives, and the need for a representation of the semantics of a sentence which can faithfully encode the desired natural-language query. We propose an approach which simultaneously addresses all of these problems [4]. Systems to date generally attempt to independently address the various aspects that make this task difficult. For example, they attempt to separately find videos that depict nouns and videos that depict verbs and essentially take the intersection of the two sets of videos [3]. This general approach of solving these problems piecemeal cannot represent crucial distinctions between otherwise similar input queries. For example, if you search for The person rode the horse and for The horse rode the person, existing systems would give the same result for both queries as they each contain the same words, but clearly the desired output for these two queries is very different. We develop a holistic approach which both combines tracking and word recognition to address the problems of unreliable object detectors and trackers and at the same time uses compositional semantics to construct the meaning of a sentence from the meaning of its words in order to make crucial but otherwise subtle distinctions between otherwise similar sentences. In order to combine trackers and word recognizers to build task-specific models which recognize target sentences, we choose representations for trackers and word recognizers such as that they share the same underlying structure. Both employ a lattice and use the same inference algorithm, dynamic programming through the Viterbi [5] algorithm. Tracker lattices [1] are constructed from the output of an object detector [2], where detections in adjacent frames are connected to each other. Detections are weighted by their object detector score and the links between detections are weighted by the a measure of the optical flow [6] between pairs of detections in adjacent frames. The optimal path through this lattice represents an object track. Words are represented as finite state machines (FSMs) which accept tracks. The path through an FSM determines if one or more tracks participate in the event or relationship described by that word. Given an input sentence and a single video we employ a dependency parser to extract the number of participants and the dependency structure which relates participants and words. For each participant we instantiate a tracker and for each word we instantiate an word recognizer. Rather than first tracking the participants and then checking if those tracks conform to the sentential query, the algorithm perform both steps simultaneously by taking the cross-product (a) The horse approached the person (b) The person approached the horse Figure 1. The top 6 hits for two sentences from a corpus of 10 Hollywood movies. In both cases, half are true positives. The fact that the results are different shows that our method encodes the meaning of the entire sentence along with which object fills which role in that