{'fraction_non_alphanumeric': 0.01597444089456869, 'fraction_numerical': 0.0, 'mean_word_length': 5.394557823129252, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data quality screening for high-resolution satellite imagery via spectral clustering

High-resolution satellite imagery data have been widely used in geoscience and remote sensing research. Dealing with data quality issue is the first and most important step before truly making use of these high-resolution images. Scientific results derived from poor-quality data can be problematic and unreliable. In this work, we propose a novel data quality screening method to discover and filter anomalous images contaminated by systematic errors in a dataset. In particular, cumulative distribution function based pairwise similarity matrix and spectral clustering are adapted to accurately identify clusters of normal and anomalous images. Using the proposed method, we have discovered abnormal images in a collection of high-resolution satellite imagery over the Arctic sea ice, which has been used as ground truth in previous melt pond studies.