{'fraction_non_alphanumeric': 0.03474812331806808, 'fraction_numerical': 0.02195363769415986, 'mean_word_length': 4.754414561260527, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Class Density and Dataset Quality in High-Dimensional, Unstructured Data

We provide a definition for class density that can be used to measure the aggregate similarity of the samples within each of the classes in a high-dimensional, unstructured dataset. We then put forth several candidate methods for calculating class density and analyze the correlation between the values each method produces with the corresponding individual class test accuracies achieved on a trained model. Additionally, we propose a definition for dataset quality for high-dimensional, unstructured data and show that those datasets that met a certain quality threshold (experimentally demonstrated to be>10 for the datasets studied) were candidates for eliding redundant data based on the individual class densities.


Introduction and Related Work
Data density is of interest in the classification domain mostly in cases where there exists a class imbalance problem, which can be quite common in many real-world data collection scenarios (for example credit card fraud detection [1] [2]). In this context, the interest is in increasing the density of the data in the minority class via synthetic sample creation [3] or in under-sampling the majority class [4]. In [5] and [6], the authors showed that generating synthetic minority class samples from data near the boundaries of the classes produced superior results as compared to completely randomly sampling from the entire minority class. In the clustering domain, data density has been analyzed to produce more intelligent clustering algorithms that do not require the prior selection of the number of clusters (as is required for k-means clustering) [7].
Studies in data density are replete in the literature for low-dimensional data, such as in those cited in the prior paragraph. However, data density is less studied in high-dimensional data. This is most likely due to the fact that distance measurements in high-dimensional space produce near uniform distances [8], a phenomenon colloquially referred to as the curse of dimensionality. As in [9], in order to reclaim meaningful differences when measuring the distances between samples, in the studies in this work, we use Uniform Manifold Projection and Approximation (UMAP) [10] to reduce high-dimensional image data to 3 dimensions.
In this work, we will be defining a density measurement that can be applied to the 3-dimensional reduction of each class in a dataset independently and then providing a study of classification accuracy after reducing the data in each class to a variety of target densities as opposed to reducing each class by the same amount as in [9].
As has been put forth and stands to reason, data quality is context dependent [11]. Most of the context in the existing literature is on highly-structured, often relational, data for which there are known domains and constraints for well defined groups (tuples) of atomic values [12] [13]. Further, There is no universally agreed upon definition of data quality, let alone a universal method for measuring it. However, there is widespread agreement on some things, such as having high levels of the following attributes: timeliness, accuracy, consistency, and completeness [14]. In the present study, we will limit our focus to quality in the sense of completeness. Only a dataset comprised of lowdimensional data with very limited domains could ever be considered complete in the most strict definition of the term. For the high-dimensional, unstructured data we will be evaluating, completeness will be demonstrated by removing data from the dataset, based on a target density threshold, while maintaining classification accuracy.

Our Contribution
Our contribution is as follows: 1. We provide a definition and method for calculating individual class densities based on 3-dimensional reductions produced by UMAP. 2. We provide a method for calculating an unstructured, high-dimensional dataset's quality, with quality specifically referring to a level of demonstrable completeness. 3. We show that for 6 out of the 7 datasets studied, datasets with a quality of > 10 could be reduced to a target density of at most 1.0 and achieve accuracy that is statistically insignificantly different from the baseline. Unique to the dataset for which this was not true (Imagenette) is the relatively low number of training samples. This combined with the large image size and the assumptions made by UMAP suggest that there is a lower bound on the number of training samples such that any elision whatsoever risks underfitting in the training.

Defining Density
Definition 1 (Class Density). The class density of a class is a measure of the aggregate similarity of the training samples available for that class.
Three candidates for the calculation of a class's density were initially considered, all of them based on the distribution of points in each dimension of any m-dimensional reduction. The three candidates are based on the minimum (see Equation 1), the maximum (see Equation 2), and the mean (see Equation 3) standard deviation of the m gaussians of the m-dimensional reduction.
For all three candidate calculations, the density is calculated for class i among all n classes where c i is the count of samples for class i. σ i are the standard deviations of the m gaussians of the m-dimensional reduction for class i.
The first three terms of each calculation are identical and are used for biasing unbalanced datasets. For unbalanced datasets, these terms create a value < 1.0 for those classes that have fewer samples than average and > 1.0 for those classes that have more samples than average. The motivation for this is to incorporate over and underrepresentation of classes into the density definition. For balanced datasets, these three terms will compute to 1.0 and thus have no effect on the broader density calculation.

Min-Derived Density Calculation
The densities were calculated using each of these calculations for each of the classes for MNIST [15], Fashion-MNIST [16], CIFAR-10 [17], CIFAR-100 [17], and Imagenette [18]. Then the individual class accuracies for every class in each of these datasets were averaged across the five trials of the baseline experiments. Then correlations were calculated, using the Pearson Product-Moment Correlation Coefficient (PPMCC), between the class densities and the class accuracies, looking for a correlation between higher accuracy and higher density. The results of this correlation Max-Derived Density Calculation Mean-Derived Density Calculation study are presented in Table 1, Table 2, Table 3, Table 4, and Table 5 (a tabulation of the correlation of the individual classes in CIFAR-100 has not been presented here due to the large number of classes). The study showed that on average, all candidate calculations showed a moderate correlation. However, the min and max candidates each had a dataset for which the correlation was negative (MNIST and Fashion-MNIST respectively). Aside from having no datasets with a negative correlation, the mean candidate also had the highest mean correlation. Notably, CIFAR-10, and CIFAR-100 had the weakest non-negative correlation for all three candidates. When excluding these datasets, the difference in the mean correlation between the min and max candidates and the mean candidate was even greater. As such, the mean candidate, Equation 3, has been chosen for the calculation of class density. all classes in the training data that had a density greater than a target density value were reduced by the number of samples needed to achieve that target density value. In order to find how many samples would need to be excluded, a binary search through each class's samples, ordered by distance from the class's centroid was performed. The binary search terminated and selected the threshold distance for inclusion after 9 iterations, which was sufficient to choose the number of samples to be included to within a margin of .05% of the total number of samples in the class.
While studying dynamic data reduction, we used the datasets for which the correlation studies were performed above (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and Imagenette) as well as two additional datasets. The first is the micro-PCB dataset introduced in [19]. Data augmentation used during the training for the micro-PCB dataset was uniform and the same strategy as used for the experiments in [20]. The second additional dataset was EMNIST-Digits [21], which is a dataset with the same format as MNIST and with the same domain of interest, namely the Hindu-Arabic numerals. The difference is that EMNIST-Digits contains exactly four times as many samples in both the training and test sets. This allows for an investigation into whether the test accuracy achieved after reducing data to a target threshold per class is sensitive to or independent of the number of samples used for training. Data augmentation used during the training for EMNIST-Digits was uniform and the same strategy as used for the experiments in [22].
Each of MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and EMNIST-Digits were subjected to this method using target density levels of 1.0, 0.9, 0.8, 0.7, 0.6, and 0.5. These target densities were chosen because they produced a similar number of excluded samples for these datasets. However, in the case of Imagenette and the micro-PCB dataset, the UMAP dimensional reduction produced values in each dimension with much larger variances (thus producing smaller densities). This is due to the fact that the UMAP algorithm assumes that the population in the higher dimensional space from which the data is sampled is uniformly distributed on some manifold. Since Imagenette and the micro-PCB dataset are both higher dimensional than the other datasets as well as having fewer samples in the training set, UMAP assumes the values of these samples in the higher dimensional space represent a much broader range of possible values for the assumed uniformity of the population. Therefore, a larger number of approximately evenly spaced target densities was chosen specifically for each of Imagenette and of the micro-PCB dataset.
For all experiments, 5 trials were conducted, and the accuracy reported is the mean accuracy of those 5 trials. Table 6 through Table 12 show the results of these experiments. Bold typeface in the Accuracy columns indicate that the accuracy matched or exceeded the accuracy of the baseline.

Experimental Results
For the MNIST dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 1.0, 0.9, 0.8, and 0.7. These target densities resulted in reducing the training dataset size by 4.4%, 7.8%, 15%, and 22.6%, respectively.
For the Fashion-MNIST dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracy of the experiment with target density 1.0 which resulted in reducing the training dataset size by 2.0%.
For the CIFAR-10 dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 1.0, 0.9, 0.8, and 0.7. These target densities resulted in reducing the training dataset size by 0.2%, 0.2%, 0.2%, and 4.2%, respectively.
For the CIFAR-100 dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 1.0 and 0.8. These target densities resulted in reducing the training dataset size by 0.4% and 2.3%, respectively, The experiment with a target density of 0.9 produced a statistically significantly superior accuracy while reducing the dataset size by 0.4%.
For the Imagenette dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 0.70, 0.65, 0.60, and 0.55. These target densities resulted in reducing the training dataset size by 0.3%, 2.4%, 5.1%, and 9.2%, respectively.
For the micro-PCB dataset, there was no statically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 0.25, 0.20, 0.15, 0.10, 0.075, and 0.05. These target densities resulted in reducing the training dataset size by 5.4%, 15.8%, 30.5%, 42.9%, and 53.2%, respectively. Of particular note with the micro-PCB dataset is that the baseline accuracy of 100% was able to be achieved with 53.2% of the training samples removed.
For the EMNIST-Digits dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 1.0, 0.9, and 0.8. These target densities resulted in reducing the training dataset size by 3.1%, 7.2%, and 13.4%, respectively. As was noted in the previous section, the UMAP dimensionality reduction algorithm makes assumptions about the distribution of the data in the higher dimensional space that results in density values that are dependent on the dataset. In order to account for this, the Mean-Derived Density Calculation (Equation 3) has been modified to include a normal-      have been elided as the purpose they served of biasing unbalanced classes does not apply to the well-balanced datasets being studied here.
In Equation 4, d i is the density of class i among all n classes. σ i and σ j are the standard deviations of the m gaussians of the m-dimensional reduction for classes i and j, respectively.
Mean-Derived and Normalized Density Calculation Table 13 through Table 18  As in the previous section, but instead using the Mean-Derived and Normalized Density Calculation (Equation 4), another set of experiments was conducted. In these experiments, the training data in each class was reduced using the Central Exclusion data reduction strategy by the number of samples necessary to ensure all classes in the training data that had a density greater than a target density value were reduced by the number of samples needed to achieve that target density value. Each of MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, Imagenette, the micro-PCB dataset, and EMNIST-Digits were subjected to this method using target density levels of 1.1, 1.05, 1.0, 0.95, and 0.9.
For all experiments, 5 trials were conducted, and the accuracy reported is the mean accuracy of those 5 trials.  Table 18:

Mean-Derived Density Calculation (Equation 3) vs. Mean-Derived and Normalized Density Calculation (Equation 4) -Fashion-MNIST
For the MNIST dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of any of the experiments with target densities. These target densities resulted in reducing the training dataset size by 4.2%, 8.0%, 11.8%, 17.2%, and 24.2%, respectively.
For the Fashion-MNIST dataset, all target densities were statistically significantly inferior to the baseline (p < 0.05). This is consistent with the results from the prior section as the highest target density resulted in removing 15% of the data whereas the only statistically insignificant data reduction using Equation 3 elided just 2% of the data.
For the CIFAR-10 dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 1.1, 1.05, and 1.0. These target densities resulted in reducing the training dataset size by 0.2%, 1.1%, and 5.4%, respectively.
For the CIFAR-100 dataset, all target densities were statistically significantly inferior to the baseline (p < 0.05). This is consistent with the results from the prior section as the highest target density resulted in removing 5.7% of the data whereas statistically insignificant data reduction using Equation 3 was only achieved when eliding 2.3% of the data.
For the Imagenette dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target density 1.1. This target density resulted in reducing the training dataset size by 1.7%. This is inconsistent with the results from the prior section as statistically insignificant data reduction using Equation 3 was achieved when eliding as much as 9.2% of the data.
For the EMNIST-Digits dataset, there was no statistically significant difference (p < 0.05) between the accuracy of the baseline and the accuracies of the experiments with target densities 1.1 and 1.05. These target densities resulted in reducing the training dataset size by 5.1% and 8.5%, respectively. After 5 trials, a statistical significance to the superior accuracy of the experiment with target density of 1.0 was narrowly missed (p < 0.07). Of worthy note for this dataset is the remarkable consistency and low variance across trials for the baseline (which all achieved the same accuracy of 99.79%) and across trials for each of the target densities. The trials for target density 1.1 produced two accuracies of 99.79% and three accuracies of 99.78%. The trials for target density 1.05 produced one accuracy of 99.79%, three accuracies of 99.78%, and the lowest accuracy of all trials of 99.76%. The trials for target density 1.0 produced two of the highest accuracies at 99.8% and three at 99.79%. The trials for target density 0.95 produced three accuracies of 99.78% and two of 99.77%. The trials for target density 0.9 produced one accuracy of 99.79%, two accuracies of 99.78%, and two accuracies of 99.77%.

Dataset Quality
The experiments detailed thus far have demonstrated that, for the datasets studied, there exists an amount of redundant data in each such dataset and that that data can be located near the centroid of a dimensional reduction of the data.
This observation inspires the possibility of a measure of the dataset quality that can be used a-priori to determine whether or not the data distributed in the lower-dimensional space is capable of leading to optimal or near-optimal classification accuracy.
Equation 5 represents an attempt to capture this measure numerically in a single value, such that the closer the value is to zero, the lower the "quality" of the dataset, or from another point of view, there exists room for improvement. In this equation, d are the densities of the classes as calculated using Equation 4. As such, the equation is simply 1 over the product of the standard deviation of the class densities and the range of density values. Thus and simply, the greater variance among the class densities, the lower the quality of the data.
Dataset Quality Calculation Table 26 shows this calculation as applied to the training datasets we have studied. MNIST, EMNIST-Digits, CIFAR-10 and the micro-PCB dataset all had values for quality > 10 and all of these datasets were shown to have had excess data in their training datasets. Fashion-MNIST and CIFAR-100 each had values for quality < 10 and both of these datasets were shown not to have had an excess of data in their training datasets. Imagenette was inconsistent with this. It is hypothesized that in the case of Imagenette, that the lower number of training samples coupled with the complexity of the features that comprise the subject matter, compared to the other datasets, dominates any ability to elide redundant data. Table 27 shows the same calculation as applied to the validation data of the studied datasets. Consistent with the training data, the validation data had values for quality > 10 for MNIST, EMNIST-Digits, CIFAR-10 and the micro-PCB dataset, and Fashion-MNIST and CIFAR-100 each had values for quality < 10.

Summary
In this work, we have put forth a definition for class density along with several methods for calculating such. Analysis of seven datasets showed that Equation 4, derived from the normalized mean standard deviations of the data points in lower dimensional space, was the appropriate choice for the calculation for density. Additionally, we put forth a definition for dataset quality in Equation 5, which showed that those datasets that met a certain quality threshold (experimentally demonstrated to be > 10 for the datasets studied) were candidates for eliding redundant data using Equation 4. The experiments showed that for all datasets except Imagenette, datasets with a quality of > 10 could be reduced to a target density of at most 1.0 and achieve accuracy that is statistically insignificantly different from the baseline. Unique to Imagenette is the relatively low number of training samples coupled with the complexity of the features that comprise the subject matter. That combined with the large image size and the assumptions made by UMAP suggest that there is a lower bound on the number of training samples such that any elision whatsoever risks underfitting in the training procedure.
Future work will attempt to incorporate a predictive quantification of the impact of image size (and thus dimensionality) and feature complexity into the quality assessment.