{'fraction_non_alphanumeric': 0.01808066759388039, 'fraction_numerical': 0.0, 'mean_word_length': 5.206896551724138, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Effects of noise pre-filtering for codebook generation in vector quantization

Encoding noisy data is a challenging problem in data compression. Recent results show that data compression and filtering are closely related. Exploiting these ideas, we propose a simple technique for vector quantization (VQ) coding of noisy images, based on pre-filtering only the training set prior to codebook design. We find that our technique results in significant improvement in image quality, comparable to that obtained by the computationally more expensive approach of pre-filtering both the training set and the images to be encoded. A preliminary justification of our results from the theory of Occam filtering is also provided.