{'fraction_non_alphanumeric': 0.029838488913222008, 'fraction_numerical': 0.010128661374212975, 'mean_word_length': 4.745283018867925, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reply to Davies

The objections which Professor Davies has raised to D and its complement, the index of consensus, merit careful consideration. It is true that, if the intervals are equal, D is a simple function of the Average Deviation from the median. From that viewpoint, it may seem than an equal interval assumption has crept into the index. There are two lines of argument which I wish to develop in defense of D: (a) that it is truly ordinal in the sense that it is invariant under transformations of a dimension provided that the order of points on that dimension is preserved; and (b) that an explicitly intervalscale treatment of D shows it to be fairly stable under various transformations ofthe basic dimension, with certain exceptions. Regarding ordinal invariance, the operations producing the numerator of D can be rearranged in the following manner. First, note what proportion of cases lies outside the median category. Next note what proportion lies beyond the first category on either side of the median category. Again, note the proportion beyond the second category either side of the median, continuing until all categories have been examined. Then simply add the proportions noted. No matter what the actual distances between category midpoints are, assuming that an objective equal-interval scale underlies the ordinal dimension, D will not be affected. Thus D is based on the percentage of cases which fall beyond identifiable, ordered points without reference to the distances between those points. The above argument may not suffice for those who cannot escape an underlying metric notion and wish to ask what happens to the Average Deviation divided by its upper limit (an interval interpretation of D) when the underlying metric changes. If the midpoints of the intervals are designated Yo, Y1, ... Ym, and the intervals between those points are ai = YT Yi-1, then different continua will be created by varying the a1. In addition to ai = c (the case for which D is above function of the Average Deviation), the following sets of values for a1 will be examined for five-category distributions: 1, 2, 3, 4; 4, 3, 2, 1; 1, 2, 2, 1 and 2, 1, 1, 2. The first wo are systematically increasing or decreasing across the continuum, the third represents end intervals only half as large as the central ones, and the last represents central intervals only half as large as the end ones. It seems unlikely that response categories to opinion or preference items will have more marked inequalities than those chosen. Nine frequency distributions have been examined for all five continua to permit comparison of values obtained. Table 1 shows the distributions and their D values. Distributions are ordered by median and dispersion, with greater dispersion in asymmetric distributions implying reater skew as well. Table 2 indicates the discrepancy between D and its interval-level equivalent for each of the unequal interval continua. It is evident that symmetrical expansion or contraction of the extreme intervals (last 2 continua) has virtually no effect on D. The largest discrepancies occur for G and H, symmetrical distributions with moderate dispersion (D = .40 and .60), and these discrepancies are not alarmingly large. It would appear that distributions B through F (skewed, with moderate dispersion; D = .20, .35, .25, .35, and .55, respectively) are somewhat more subject to systematic increase or decrease in continuum intervals. The worst of these is C, which is highly skewed, yet covers most of the possible range. Symmetrical distributions G, H and I, however, are totally unaffected by these changes in interval width.