{'fraction_non_alphanumeric': 0.025060630557801132, 'fraction_numerical': 0.002425222312045271, 'mean_word_length': 5.284263959390863, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
New feature preserving noise removal algorithm based on the discrete cosine transform and the a prior knowledge of pixel type

In this paper, a new corrupted-pixel-identification (CPI) based estimation filter is presented. The method is especially useful for filtering clustered noise. After the corrupted pixels are identified by the CPI algorithm, the noisy subimage centered on a corrupted pixel is transformed into its discrete cosine transform (DCT) domain where the transformed subimage is approximated by its DC coefficient only. With the knowledge of the number of corrupted pixels in the subimage from the CPI algorithm, an estimation of the noise distribution can be made, from which the DC coefficient of the restored subimage can be determined. Hence, noise filtering is achieved. From the experimental results, we can show that the CPI-based estimation filter has three desirable characteristics: (1) it has superior noise removal performance over the conventional median filter and CPI-based median filter in filtering clustered noise; (2) it has good feature preserving property (better than conventional filters); and (3) the computing speed of the filter is almost three times faster than the conventional median filter.