{'fraction_non_alphanumeric': 0.01597444089456869, 'fraction_numerical': 0.0019169329073482429, 'mean_word_length': 5.6075949367088604, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Quantification of Code Regularity Using Preprocessing and Compression

Complexity metrics are useful to identify potentially problematic code. But there is little agreement regarding which complexity metrics should be used and exactly what should be measured, partly because many different factors influence code complexity and comprehension. Code regularity has recently been identified as another such factor, which may compensate for other complexity factors, especially in long functions with high cyclomatic complexity. Given that regularity consists of repeated code structures, it has been suggested to measure regularity by compressing the code with standard text compression tools. But such compression can be done in many ways. We compare five widely available compression tools (LZ77, gzip, LZMA, bzip2, and bicom) and four levels of preprocessing the code (using the code as is, or reducing it to a skeleton of keywords and possibly some formatting). The comparison is done in terms of how the different combinations discriminate between different functions, how they correlate with human perceptions of complexity, and how well they handle relatively short functions. The results show that different combinations of compression tool and code preprocessing lead to significantly different levels of discrimination and correlation with human perceptions, and in addition some combinations are extremely bad in handling many functions and should be avoided. Our recommendation is to use gzip or bicom on a code skeleton containing keywords and formatting.