{'fraction_non_alphanumeric': 0.020134228187919462, 'fraction_numerical': 0.0009587727708533077, 'mean_word_length': 5.484472049689441, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated Hierarchical Density Shaving and Gene DIVER

A key application of clustering data obtained from sources such as microarray, protein mass spectroscopy, and phylogenetic profile is the detection of functionally related genes. Typically, only a small number of functionally related genes cluster into one or more groups, and the rest need to be pruned. For such situations, we present Automated Hierarchical Density Shaving (Auto-HDS), a framework that consists of a fast, hierarchical, densitybased clustering algorithm and an unsupervised model selection strategy. Auto-HDS can automatically select between clusters of different densities, present them in a compact hierarchy and rank individual clusters using an innovative stability criteria. Our framework also provides a simple yet powerful 2-D visualization of the hierarchy of clusters that can be very useful for further exploring the dense clusters in high-dimensional datasets. We present results on Gasch and Lee microarray datasets to show the effectiveness of our methods.