{'fraction_non_alphanumeric': 0.014071294559099437, 'fraction_numerical': 0.0009380863039399625, 'mean_word_length': 5.276470588235294, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Outlier detection in hyperspectral imagery using closest distance to center with ellipsoidal multivariate trimming

In this paper we examine the efficacy of using the closest distance to center algorithm in conjunction with ellipsoidal multivariate trimming (MVT) to find outliers in a hyperspectral image. MVT is applied here as a global anomaly detector on images that are pre-processed into clusters using a technique called X-means. Under the assumption that there are no more than 5% outliers in any given cluster set, we develop a method, based upon principal component analysis pre-processing, to create a flexible threshold for determining the percentage of data to retain with MVT. Using a retention percentage that more adequately reflects the actual number of outlier-free observations allows one to form estimates of the mean and covariance matrix that more effectively decrease the effects of swamping and masking as compared to using a set percentile for retention. These ideas are tested against real and synthetically generated hyperspectral imagery.