{'fraction_non_alphanumeric': 0.024242424242424242, 'fraction_numerical': 0.008484848484848486, 'mean_word_length': 5.2105263157894735, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Subject Areas by Using Unsupervised Observation of Most Informative Terms in Text Databases

Much work has been done on automatic topic detection using learning by example techniques (1) but they are confined to assignment of terms onto a predetermined classes using some algorithms which are trained on some manually classified documents and most of them identify only text boundaries. But in the present we do not use any prior information for subject area identification. Here we scan most informative/prominent words from the corpus (2) and they are grouped using learning by observation techniques (3) and some of the well known distance measures (4,5) for subject area identification. We may use similarity measures like total divergence to the average (6), Bhattacharya co-efficient (7) for term similarities.