{'fraction_non_alphanumeric': 0.083373963920039, 'fraction_numerical': 0.006338371526084837, 'mean_word_length': 5.1437125748503, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On Limited Memory Universal Classification of Individual Sequences

We discuss a device called "classifier" (or discriminator) that observes an individual "training" sequence of length of m letters, X<sup>m</sup> <sub>1</sub>. The classifier's task is to examine individual test sequences of length N and decide whether the test N-sequence has the same features as those that are captured by the training sequence, or is sufficiently different, according to some appropriate criterion. No a-priori information about the test sequences is available to the classifier, aside from the training sequence. A universal classifier d(N,Z<sup>N</sup> <sub>1</sub> isin A<sup>N</sup>) for N-vectors is a mapping from A<sup>N</sup> onto {0,1}. Upon observing Z<sup>N</sup> <sub>1</sub> , the classifier declares Z<sup>N</sup> <sub>1 </sub> to be "similar" to the training sequence X<sup>m</sup> <sub>1</sub>, iff d(N,Z<sup>N</sup> <sub>1</sub>) = 1. A trivial way to proceed is to compare the test sequence X<sup>N</sup> <sub>1</sub> with each of the distinct N-vectors that appear in the training sequence X<sup>m</sup> <sub>1</sub>. However, this procedure calls for a storage-space complexity that may grow exponentially with N. It is demonstrated that a particular universal context-tree classifier with a computational and storage complexity that is linear in N is essentially optimal. Another task for an N-sequence classifier is to decide whether two N-sequences Y<sup>N</sup> <sub>1</sub> and Z<sup>N</sup> <sub>1</sub>, or more, are similar to each other in the sense that each of the N-sequences is similar (as defined above) to the same unknown training sequence which is not available, (e.g "do these N-sequences sequences have a "common ancestor" - a frequent topic in computational biology). It is demonstrated that such an hypothesis may be essentially optimally tested based on the same context-tree classifier. This contributes a theoretical "individual sequence" justification for the probabilistic suffix tree approach in computational biology