{'fraction_non_alphanumeric': 0.02328863796753705, 'fraction_numerical': 0.0, 'mean_word_length': 5.504587155963303, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Constrained and recursive hierarchical table-lookup vector quantization

This paper presents techniques for the design of generic constrained and recursive vector quantizer encoders implemented by table-lookups. These vector quantizers include entropy-constrained VQ, tree structured VQ, classified VQ, product VQ, mean-removed VQ, multi-stage VQ, hierarchical VQ, nonlinear interpolative VQ, predictive VQ and weighted universal VQ. Our algorithms combine these different VQ structures with hierarchical table-lookup vector quantization. Thus the full-search encoder in the different VQ structures is replaced by a table-lookup encoder, which approximates the search, but the codebook structure and decoder are the same. In these table-lookup encoders, input vectors to the encoders are used directly as addresses in code tables to choose the codewords. In order to preserve manageable table sizes for large dimension VQs, we use hierarchical structures to quantize the vector successively in stages. Since both the encoder and decoder are implemented by table-lookups, there are no arithmetic computations required in the final system implementation. To further improve the subjective quality of the compressed images we use block transform based table-lookup vector quantizers with subjective distortion measures. There is no need to perform the forward or reverse transforms as they are implemented in the tables.