{'fraction_non_alphanumeric': 0.01998001998001998, 'fraction_numerical': 0.003996003996003996, 'mean_word_length': 5.423076923076923, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards a general model of non-random sampling and the impact on population correlation: generalizations of Berkson's Fallacy and restriction of range.

A long history of restriction of range research has developed in the psychometric literature under the assumption of strict truncation. A conceptualization of the general problem of non-random sampling is proposed leading to a model of which both restriction of range and Berkson's Fallacy (Berkson, 1946) are special cases. Cast in the framework of unequal sampling probabilities operating separately on the variables of interest, two major forms of the model are investigated: an additive and an interactive (multiplicative) form. The influence of unequal selection probabilities on population values of the bivariate normal correlation (ρ) is described and findings from other research on restriction of range are shown to be special cases of these results. Implications for inferences from data and evaluation of research designs are discussed.