{'fraction_non_alphanumeric': 0.014612452350698857, 'fraction_numerical': 0.0, 'mean_word_length': 5.225296442687747, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Iterative unified clustering in big data

We propose a novel iterative unified clustering algorithm for data with both continuous and categorical variables, in the big data environment. Clustering is a well-studied problem and finds several applications. However, none of the big data clustering works discuss the challenge of mixed attribute datasets, with both categorical and continuous attributes. We study an application in the health care domain namely Case Based Reasoning (CBR), which refers to solving new problems based on solutions to similar past problems. This is particularly useful when there is a large set of clinical records with several types of attributes, from which similar patients need to be identified. We go one step further and include the genomic components of patient records to enhance the CBR discovery. Thus, our contributions in this paper spans across the big data algorithmic research and a key contribution to the domain of heath care information technology research. First, our clustering algorithm deals with both continuous and categorical variables in the data; second, our clustering algorithm is iterative where it finds the clusters which are not well formed and iteratively drills down to form well defined clusters at the end of the process; third we provide a novel approach to CBR across clinical and genomic data. Our research has implications for clinical trials and facilitating precision diagnostics in large and heterogeneous patient records. We present extensive experimental results to show the efficacy or our approach.