{'fraction_non_alphanumeric': 0.028205128205128206, 'fraction_numerical': 0.0, 'mean_word_length': 4.255892255892256, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On coding for data analytics: New information distances

Distance plays a vital role in many applications of data analytics. In this paper, the concept of distance between any two data objects X and Y is addressed from the perspective of Shannon information theory. Consider a coding paradigm where X and Y are encoded into a sequence of coded bits specifying a codeword (or method) which would in turn convert Y into X, and X into Y such that both the distortion between X and X and the distortion between Y and Y are less than or equal to a prescribed threshold D. Given a class C of coding schemes within the coding paradigm, the information distance RC (X, Y, D) between X and Y at the distortion level D is defined as the smallest number of coded bits afforded by coding schemes from C. For two important classes C, RC (X, Y, D) is shown to be indeed a pseudo distance in some sense; it is further characterized or bounded. When C is the class of so-called separately precoded broadcast codes, it is shown that for any stationary, totally ergodic sources X and Y, RC (X, Y, D) is equal to the maximum of the Wyner-Ziv coding rate of X with Y as side information and the Wyner-Ziv coding rate of Y with X as side information. In the general case where C consists of all codes within the coding paradigm, upper and lower bounds to RC (X, Y, D) are established, and are further shown to be tight when X and Y are jointly Gaussian. The distance RC (X, Y, D) generalizes the notion of information distance defined within the framework of Kolmogorov complexity.