{'fraction_non_alphanumeric': 0.01040582726326743, 'fraction_numerical': 0.0, 'mean_word_length': 5.287581699346405, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Entropy Based Measure Functions for Analyzing Time Stamped Documents

Measure functions that assign numeric values to keywords to capture their significance in a document set play a crucial role in the construction of a time decomposition of a document set. In this paper, we define two measure functions based on the notion of entropy. The interval entropy measure function identifies time intervals that have non-uniform keyword distributions and assigns high measure function values to keywords with high relative occurrence frequency in that time interval. The keyword entropy measure function similarly identifies keywords that have non-uniform occurrence frequency over time. The measure functions are applied to construct several time decompositions of a subset of documents from the TDT-Pilot corpus and the Enron Email data set. The results indicate that the measure functions are highly effective in capturing the temporal content of the document set.