{'fraction_non_alphanumeric': 0.02049530315969257, 'fraction_numerical': 0.0, 'mean_word_length': 5.23404255319149, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Character correlation and its use for identification.

A method is presented for correlating phylogenetic characters through cladistic analysis. It extends the use of phylogenetic datasets for diagnostic purposes. It improves matrix-based identification tools by predicting novel character-state combinations that were not observed when the key was constructed. By interpreting homoplasy as analytical error, hypothetical character-state combinations are tested for the homoplasy that they would add to the shortest tree(s). The correlation is equal to the homoplasy summed across all state combinations, divided by a maximum possible value. The results depend on uncertainty about the sequence of state transitions and their overlap among characters. A correlation index r is proposed for sets of non-additive characters; it is a kind of multiple-regression value, and its ensemble value R is a statistic of a whole matrix. This approach can be used to select sets of the best "proxy" characters to substitute for unobservable characters of interest. The concept can be extended to continuous characters. Worked examples are given with datasets of various insect orders.