{'fraction_non_alphanumeric': 0.017810026385224276, 'fraction_numerical': 0.0, 'mean_word_length': 5.742222222222222, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Linear Poisson Models: A Pattern Recognition Solution to the Histogram Composition Problem

The use of histogram data is ubiquitous within the sciences and histograms composed of linear combinations of sub-processes are common. The statistical properties of recorded frequencies are well understood, i.e. Poisson statistics can model the probabilities of observing discrete events, from which an in-depth theoretical analysis of histogram components and their errors can be derived. Despite this, linear decomposition techniques such as Independent Component Analysis, Principal Component Analysis, and their variants, have largely focused on continuous data, with little or no account of error characteristics on estimated parameters. We argue that the properties of histograms, i.e. Poisson, non-negative discrete frequencies, requires a linear decomposition technique for making quantitative measurements. We also argue that a solution appropriate for scientific analysis tasks must involve a good understanding of noise so that uncertainties in measurements can be reported to end users. We present an approach to the analysis of histograms capable of summarising data in terms of Probability Mass Functions, weighting quantities and associated covariance matrices which we believe is suitable for scientific applications. The end result is a quantitative pattern recognition system capable of learning distributions from training data, then estimating the quantity of similar distributions in new incoming data.