{'fraction_non_alphanumeric': 0.028416779431664412, 'fraction_numerical': 0.0, 'mean_word_length': 5.218487394957983, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A new information theoretic clustering algorithm using k-nn

We develop a new non-parametric hierarchical information theoretic clustering algorithm based on implicit estimation of cluster densities using k-nearest neighbors (k-nn). Compared to a kernel-based procedure, our k-nn approach is very robust with respect to the parameter choices, with a key ability to detect clusters of vastly different scales. Of particular importance is the use of two different values of k, depending on the evaluation of within-cluster entropy or a cross-cluster cross-entropy in order to obtain the final clustering. We conduct clustering experiments, and report promising results, focusing in particular on the proposed algorithm's robustness to scale.