{'fraction_non_alphanumeric': 0.010062893081761006, 'fraction_numerical': 0.0, 'mean_word_length': 5.578512396694215, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Use of dynamic discrimination values in a document retrieval system

The use of discrimination values as a term weighting function in document retrieval systems is examined. It is shown that regular discrimination values are too costly to compute after every update to the data base. Dynamic discrimination values that are easy to update are defined for use as approximations to regular values. Experiments are performed comparing regular vs. dynamic discrimination values. Actual user queries from an operational data base are used to evaluate dynamic discrimination values in a production environment. Generalized forms of normalized recall and precision are used as evaluation measures. Retrieval results indicate statistically significant improvements using dynamic discrimination weighting.