{'fraction_non_alphanumeric': 0.01313755795981453, 'fraction_numerical': 0.0, 'mean_word_length': 5.709844559585492, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Trainable News Broadcast Boundary Identification Using Feature Density

In order to find relevant sections of news broadcast video, search engines must provide users with precise news results, such as specific news stories related to a user’s query. This requires that boundaries between different stories and commercials be automatically identified in the broadcast and the content between boundaries be categorized. Previous attempts to segment broadcasts rely heavily on topic classification of text or occurrence of various audiovisual features. This paper presents a novel trainable method of identifying the boundaries using the density of multimodal features, like logo detection, speaker identification and scene-cuts, in the broadcast. The training algorithm first identifies the features most indicative of a boundary by selecting the features with the greatest difference in density near boundaries and density across the broadcast. The boundary identification algorithm then marks boundaries whose feature densities match the density patterns of the most important features found during training. This new method was tested on video data from multinational and multilingual sources, such as CNN Headline News and Al-Jazeera, and reliably identified real boundaries with high recall.