{'fraction_non_alphanumeric': 0.011093502377179081, 'fraction_numerical': 0.0, 'mean_word_length': 5.221674876847291, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Nonparametric Assessment of Atypicality

Atypicality indices have been used recently in a number of applications to assess how unusual are new observations relative to a given population. We introduce a new nonparametric estimator of an index of atypicality in which kernel density estimation is employed in the original definition, due to Aitchison, which becomes a functional of the unknown density. This estimator overcomes difficulties due to bias that arise if the kernel density estimator is simply plugged in to this functional. Detailed arguments are used to determine the pointwise bias and mean-squared error properties of the estimator, which are shown to depend on the location of the new observation, and the optimal rate of convergence of the bandwidth is determined. Rather surprisingly, it emerges that this optimal rate is the same as that which optimises performance of the kernel estimator of the unknown density and so we recommend that a standard bandwidth selector is used for estimating atypicality using the new estimator. The numerical behaviour of the new estimator is studied using simulation, illustrating that its behaviour depends on the location of the new observation, and applied in the assessment of atypicality using real data.