{'fraction_non_alphanumeric': 0.03249827997895504, 'fraction_numerical': 0.019992715204986036, 'mean_word_length': 4.78553032076797, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CE-Dedup: Cost-Effective Convolutional Neural Nets Training based on Image Deduplication

Attributed to the ever-increasing large image datasets, Convolutional Neural Networks (CNNs) have become popular for vision-based tasks. It is generally admirable to have larger-sized datasets for higher network training accuracies. However, the impact of dataset quality has not to be involved. It is reasonable to assume the near-duplicate images exist in the datasets. For instance, the Street View House Numbers (SVHN) dataset having cropped house plate digits from 0 to 9 are likely to have repetitive digits from the same/similar house plates. Redundant images may take up a certain portion of the dataset without consciousness. While contributing little to no accuracy improvement for the CNNs training, these duplicated images unnecessarily pose extra resource and computation consumption. To this end, this paper proposes a framework to assess the impact of the near-duplicate images on CNN training performance, called CE-Dedup. Specifically, CE-Dedup associates a hashing-based image deduplication approach with downstream CNNs-based image classification tasks. CE-Dedup balances the tradeoff between a large deduplication ratio and a stable accuracy by adjusting the deduplication threshold. The effectiveness of CE-Dedup is validated through extensive experiments on well-known CNN benchmarks. On one hand, while maintaining the same validation accuracy, CE-Dedup can reduce the dataset size by 23%. On the other hand, when allowing a small validation accuracy drop (by 5%), CE-Dedup can trim the dataset size by 75%.


Introduction
Empowered by the large volume of data, deep neural networks have achieved impressive results in natural language processing, computer vision, and reinforcement learning. In particular, Convolutional Neural Networks (CNNs) achieve near human-level performance on a variety of vision-based tasks. This success is primarily attributed to the enormous volume of image data. Well-known datasets such as CIFAR100 [11], ImageNet [6], COCO [15] have images ranging from several thousand to millions.
In general, with a larger volume of datasets, the trained network has higher accuracy and more robustness. However, the quality of the datasets has not been assessed [13]. There is inevitably redundant information in the millionscale datasets. For example, the SVHN dataset contains cropped digits from street view house number plates. It is reasonable that exists repetitive images with similar plate materials and the same printed digits. These redundant images (we refer to as near-duplicates) bring two negative impacts.
Excessive resource consumption. Storing these redundant images would consume huge resources and costs. For instance, ImageNet [6] holds over 1.2 million images, taking up 150 GB of disk space. This situation is aggravated when training is distributed, as each cluster needs to hold a copy of the dataset. To train a moderate CNN, ResNet50 [9] in parallel, [8] uses a server of over 250 clusters, which amounts to 37.5 TB data in total. Training on these datasets will also impose excessive resources and costs. Microsoft and OpenAI [24] have worked together to develop a supercomputer dedicated to the GPT-3 model training [4]. The supercomputer consists of over 285,000 CPUs, 10,000 GPUs, costing $4.6 million only for the training process, not to mention the carbon emissions caused by the training and wireless data transmission [27].
Little to no accuracy and robustness improvement. CNNs learn little information from these redundant images. For example, Fig. 1(a) shows 8 pairs of near-duplicate (ND) images existing in the CIFAR10 and CIFAR100 datasets. The images in each pair can be derived from each other by either color or geometric transformations, which has been the de facto standard as data augmentation approaches in pre-processing. Thus, storing and training those ND images would have limited contribution to the accuracy and robustness improvement. What's worse, the ND images may be split into training and testing sets without awareness. We observe that 6 out of the 8 ND image pairs in Fig. 1(a) are separated in the training and testing sets (the training and testing sets are made officially from [20]). Fig. 1(b) shows samples of ND images with the file names from the original ImageNet [6]. As a result, the trained CNNs achieve high accuracy on the testing set. However, they will suffer drastic accuracy drops and even fail in practical deployment. Such issues are hard to detect in real-life applications.
It is therefore essential to review the huge amount of data and remove the redundant images. Image deduplication is a popular technique widely used by cloud service providers to recognize and remove duplicate images. To find duplicates, many hashing-based image deduplication algorithms are proposed. The main concept is to encode images into hashes, then find the duplicates by calculating the hamming distances between all pairs of hashes. If a distance is less than a pre-defined threshold, the two images are considered duplicates. Thus, one copy can be discarded. After removing the duplicated images, the storage size for the dataset is significantly reduced while the discriminant features in the images are still kept.  Figure 1: Sampled ND image pairs from the CIFAR(a) and ImageNet(b) datasets. In each pair, the images have high visual similarity where each image can be derived with certain transformation from each other. Original file names associated with each image are only shown for ImageNet due to the reason that Cifar10 and Cifar100 do not provide the individual file names.
We consider incorporating image deduplication into a CNN-training pipeline, which to our knowledge, is the first of proposing such strategy. Deduplication performance is controlled by a pre-defined threshold comparing the pair-wise hamming distances. In general, a small threshold brings a slightly decreased dataset. Hypothetically, CNNs trained on such a dataset should have test accuracies comparable to those trained on the original dataset. While a large threshold enables a greatly size-reduced dataset, but most likely would result in degraded test accuracy. Ideally, one would like to find the optimal threshold that reduces the dataset volume while also maintains a high test accuracy.
Towards this end, we propose CE-Dedup, a framework that combines hashingbased image deduplication algorithms with CNN classification tasks to achieve cost-effective CNN training.
In summary, we make the following contributions: • We propose CE-Dedup which, to our knowledge, is the first approach that combines near-duplicate images reduction with image classification tasks. It incorporates a hashing-based image deduplication approach with CNN training. By controlling the distance threshold, one can balance between the largest data size reduction and the highest accuracy retain. The framework CE-Dedup will be made publicly available.
• We analyze and characterize the performance impact brought by hashingbased image deduplication under different setups. Through comprehensive experiments on large-scale datasets, we show the SVHN dataset can be reduced by over 23% while maintaining the same accuracy as the original one. When the SVHN is deduplicated by over 75%, we observe a 5% accuracy drop. Figure 2: A general workflow of hashing-based image deduplication. Each image is processed and converted to a 64-bit integer hash. The hash value is compared with existing hash values using hamming distance. If there exists a distance smaller than the threshold, the input is considered as a duplicate. Otherwise, the image is unique, and the image hash is added to the hashing table.

Framework Overview
The Algorithm 1 shows the pipeline of CE-Dedup. It consists of two stages, image deduplication, and CNN training.
In stage 1, we apply various advanced hashing-based image deduplication approaches on several commonly used large-scale datasets. With different threshold settings, we assess the deduplication efficacy of each approach from aspects of 1) hashing time, 2) memory consumption, and 3) average duplicates found per image.
In stage 2, we train CNNs on deduplicated datasets. The 1) validation accuracy, and 2) training time derived from the original datasets and the deduplicated datasets are compared to quantify the effectiveness of CE-Dedup. We also explore the duplicates that exist in these large-scale datasets and how would deduplication help boost the performance of CNNs.
In the following two sections (Section 3 and 4), the details of each stage will be presented respectively. Then Section 5 introduces the datasets used and CNN training setups in extensive experiments. In Section 6, the performances of deduplication and CNN training are evaluated. Section 7 reviews related works to hashing-based image deduplication techniques. Finally, Section 8 concludes the proposed framework with an in-depth discussion.

Deduplication Workflow
A general workflow for hashing-based image deduplication is shown in Fig 2. It involves five steps as following, • Pre-processing. RGB Images are converted to grey scales, followed by down-sampling to reduce the resolution. Most algorithms resize images to 8 × 8 pixels.

Algorithm 1 CE-Dedup pipeline
Stage 1 -Image Deduplication: Convert images to grayscale; Resize images to 8 × 8; Binarize images based on a specified hashing algorithm; Flatten each hash matrix into a 1D vector; Calculate the Hamming distance D i,j for each hash vector pair i, j, and compare with a pre-defined threshold th; Stage 2 -CNN training: Perform data augmentation on the deduplicated dataset, including, • random color perturbation; • random rotation; • random affine transformation; • random crop; Train the CNN using cross-entropy loss; Record the Validation accuracy; • Hashing. Each pixel in the down-sampled images is converted from the original unsigned 8-bit integer ranging from 0 to 255, to a single binary value, either 0 or 1. This resulting 8 × 8 binary matrix is subsequently flattened into a 64-bit integer, which represents the original image.
• Storing. Hash values are stored in a table for an efficient query. Since each image hash string takes up 113 bytes ram memory, million-scale dataset such as ImageNet costing over 100Mb can easily be handled by modern computer. Commonly used data structures include KD-Tree, hierarchical K-means, and k-nearest neighbor.
• Distance Calculating. To compare the similarities between any two image hashes, the Hamming distance is used to compute the number of matched bits.
• Deduplication. Only the unique image is kept and duplicates are discarded (as shown in Fig 1, which image is kept would not be significantly different for downstream classification task). A threshold th (th ∈ [0, 63]) that allows for the maximum number of different bits in duplicates is predefined 1 . The value of 0 means that duplicates must have all bits to be the same. And the value of 12 indicates that duplicate images have less than or equal to 12 same hash bits.

Hashing Algorithm Review
Image hashing analyzes image content and then creates a value based on its unique identification details. As shown in Fig 2, a hash function is applied to the input image, and a hash value is computed based on the visual appearance of the image. In image deduplication, similar pictures should have similar hashes as well. In this way, ND images can be detected by simply compare hash differences. This paper considers four widely used hashing techniques, AHash, DHash, PHash, and WHash. Their results for stage 1 including hashing time T ime Hash , duplicates found Dups, reduced dataset size Size, memory cost M em, as well as results for stage 2 including CNNs training time on the deduplicated datasets T ime T rain and validation accuracy Acc are presented. The hashing code is primarily based on [7,10].

AHash
The Average Hashing (AHash) algorithm [5] removes high frequencies and details by down-sampling image to pixels with size N × N (8 × 8 in this paper). The RGB image is subsequently converted to grey scale and the average pixel value P mean is calculated. To convert each pixel P val to a bit value P bit , P val is compared to the P mean and assigned 1 if greater than P mean otherwise 0.
Then the 8 × 8 binary matrix is flattened into 64-bit integer to obtain the final hash. AHash is a simple and fast hashing algorithm. However, directly comparing with the mean may not yield stable results.

DHash
The Difference Hashing (DHash) algorithm [5] firstly resizes image with resolution of N × (N + 1) (8 × 9 in this paper). The RGB image is then converted to grey scales. To binarize the pixels, DHash compares the adjacent pixels. Specifically, each pixel is set to 1 if its pixel value is larger than the right pixel, otherwise 0.
This operation generates an 8 × 8 binary matrix. DHash algorithm has an improvement over AHash by comparing adjacent pixel pairs, thus more local patterns can be reserved.
(I) (II) Figure 4: two groups of ND images from the original UKBench dataset(I) and augmented UKBench(II) with color perturbation, perspective transformation, random rotated crop, and random erasing.

PHash
The Perceptual Hashing (PHash) algorithm [30] firstly resizes the image to pixels with a resolution of 32 × 32. Then, it applies Discrete Cosine Transform (DCT) [1] to turn spatial RGB values into a collection of frequencies and magnitudes. To remove the high-frequency components, only the top-left 2dimensional matrix with size of 8 × 8 is kept. The mean value of the matrix is computed to generate the binary matrix, and further flattened to a 64-bit integer as the hash representation. PHash is extensively adopted due to its efficient DCT implementation, which is more robust to certain transformations.

WHash
The Wavelet Hashing (WHash) algorithm [26] is similar to PHash, except that it uses Discrete Wavelet Transform (DWT) [25] to convert spatial RGB values to frequencies and magnitudes.

CNN Training
CNN is designed to process data with a rigid grid pattern, such as 1-D signal, 2-D images, and 3-D Voxels. It processes the spatial features from low to highlevel hierarchical order. Fig 3 introduces a general workflow of CNN training. Please refer to the paper [17] for more details.
A CNN takes an image as the input and processes it layer by layer through feature transformation to output a probability distribution. Cross-entropy loss is adopted to optimize the probability distribution toward the ground truth by back-propagating the loss to the parameters of the CNN. For better generalizability, data augmentation techniques are commonly adopted to pre-process input images. Those techniques include random color perturbation, random rotation, random affine transformation, and random crop [3].
• CIFAR10 The CIFAR10 dataset contains a total of 60000 images from 10 classes of natural objects. Classes include the airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. It is a well-known benchmark dataset used by many classification algorithms.
• CIFAR100 The CIFAR100 dataset comprises 100 classes with 600 images in each class. Classes are a wide range of natural objects from animals such as dogs, whales to objects such as trains, motorcycles, etc.
• SVHN The Streat View and House Number (SVHN) dataset has cropped digits of images taken from house numbers. It has a total of 99287 images. Each image has a digit from 0 to 9 as the ground truth.
• UKBench The UKBench dataset is a well-known benchmark used in image deduplication. In the dataset, every four ND images are different viewpoints taken from one object and organized as one group. There are a total of 2550 groups accounting for 10200 images in the dataset. Fig 4(I) shows one group from the dataset.
• AUG-UKBench We augment the UKBench dataset and construct the AUG-UKBench dataset. One image of each group from UKBench is chosen, followed by the common data augmentation operations including random color perturbation, random perspective transformation, resized crop, and random erasing, respectively. This results in a new group containing five ND images (see Fig 4(II)). We use Torchvision to generate the new dataset.
To evaluate the performance of the hashing-based image deduplication algorithm, we conduct experiments on the CIFAR10, CIFAR100, and SVHN, respectively. For each dataset, the metrics of hashing time, memory used, and duplicates found for different algorithms are compared under various threshold settings. UKBench and Aug-UKBench are used to evaluate the deduplication performance under different thresholds.
In the experiments of CNN training and classification, the CIFAR10, CI-FAR100, and SVHN datasets after the deduplication are used. For each dataset, the performance of different algorithms are compared from the aspects of training time and validation accuracy.

CNN Training
The ResNet50 [9] architecture is employed to train image classifications on the deduplicated training sets. CNNs are trained to classify images from 10 classes in the CIFAR10 datasets and 100 classes in the CIFAR100 dataset. For SVHN, the models are trained to classify 10 digits from 0 to 9. We adopt the official train and validation split from Torchvision Datasets [21].
During the training process, images are resized to 32 × 32 pixels to be compatible with the original dimensions. The parameters of the CNNs are trained using backpropagation with cross-entropy loss. We use Adam optimizer with betas 0.9 and 0.999, respectively, a weight decay of 0, and epsilon of 10 −8 . In each training epoch, a mini-batch of 64 images is configured. The initial learning rate is set to 0.01, with a learning rate decay of 10 every 40 epochs. A total of 100 epochs are used to train each model.
The implementation is based on Pytorch [19]. We use servers with hardware configured with CPU Intel i9-9900K clocked at 3.60GHz, 32G ddr4 RAM, and Nvidia RTX 2080Ti GPU. For each deduplicated dataset, we train five ResNet50 with each using a different initialization seed.

Image Deduplication Performance
We start by evaluating the deduplication efficacy under different thresholds. We plot the median ND images found for each hashing algorithm when threshold varies from 0 to 50. As shown in Fig 5,6, when the threshold is above 12, found ND images are over 150 for AUG UKBench. Given the property of UKBench and AUG UKBench, ND images should be 4-5, indicating an optimal threshold be 6 to 8. For an in-depth comparison, we use thredholds from 0 to 12 for the rest of the experiments.
Next, We evaluate the performance of different deduplication algorithms presented in Section 3.2. For each hashing algorithm, we record the hashing time, the memory cost, and the mean NDs for each image. In detail, T ime Hash is the mean time with standard deviation in seconds, M em is the mean memories with standard deviations in megabytes, and Dup is the mean number of ND images detected for each input.
In the CIFAR10 dataset (Table 1), when th increases from 0 to 12, PHash and DHash detect duplicates of 1 to 1.64 and 1.90, respectively. When setting th to 12, AHash and WHash yield candidates of up to 34 and 38.32, respectively. Due to the similar data distribution for CIFAR100, we obtain comparable results. The result for SVHN is in Table 2. There are generally more duplicates in the datasets. When setting th to 12, PHash detects 3.79 duplicates, and DHash, AHash, WHash detects duplicates of 26.83, 52.32, and 57.30, respectively. In Fig 7, we plot four exemplar duplicates detected by each hashing algorithm in the SVHN dataset with th setting to 10. The top left image in (a) -(d) is the same for a fair comparison. It shows that PHash and AHash are more effective in detecting duplicates with similar visual displays. This confirms our hypothesis that SVHN dataset contains many NDs.
In summary, PHash and DHash achieve the most stable performance under   various threshold settings. For the hashing time, PHash, DHash, and AHash prove to be more efficient than WHash. However, WHash detects a significantly escalated number of duplicates when the threshold increases. Further, to achieve a larger deduplication ratio under the same threshold, WHash and AHash are better candidates by detecting more ND images.

CNN Training Performance
We study the effectiveness of deduplicated datasets in the CNN training application. We train CNNs on the deduplicated CIFAR10, CIFAR100, and SVHN datasets. Training and validation split is from the official Torchvision Datasets [21]. The number of images after the deduplication in the training dataset D train and in the validation dataset D val are reported, respectively. On each training dataset, we train ResNet50 [9] on 5 random initializations. The mean and standard deviation of validation accuracy Acc and training time T ime train are calculated. The performance in the original un-processed dataset is also presented. Table 3 shows the results for the deduplicated CIFAR10 dataset. PHash and DHash demonstrate a consistent validation accuracy of 0.81 when using a threshold up to 4. The image number of the training dataset is reduced to 49709, and the validation data is slightly down to 9940. We observe a speedup of 200 seconds in training time compared with the original dataset. AHash and WHash algorithms exhibit a small accuracy drop but detect more duplicates overall. Specifically, after the deduplication with a threshold of 12, PHash and DHash have over 40000 images left in the training set, whereas AHash and WHash have less than 12000 images.
Similar results for deduplicated CIFAR100 dataset are obtained. PHash and DHash maintain a consistent accuracy from 0.43 to 0.40 when the threshold increases from 0 to 12. And the dataset size ranges from over 49900 to 42969 and 38577, respectively. In contrast, AHash and WHash have the largest fluctuation with accuracy drops from 0.43 to 0.12 lowest, and dataset size varies from about 49000 to over 10000. Table 4 illustrates the results for the deduplicated SVHN datasets. We can see PHash achieves the best overall accuracy. Validation accuracy maintains 0.92 even when the threshold is set to 10. The dataset is reduced to 59584 images, and the training time is reduced to only 25%. AHash, DHash, and WHash get comparable results on the dataset size and training time. But WHash has the overall lowest accuracy, especially when the threshold is set to 12. In summary, as suggested from Fig 5,6, a proper deduplication threshold for safely removing ND images should be set between 6 to 8. Futher experiments from the CNN-based perspective suggest that PHash and DHash again retain relatively stable accuracy when the threshold increase from 0 to 6. The deduplicated dataset is reduced by 77% largest, with a 30% speedup in training time. We therefore conclude that using PHash and DHash with the threshold set to 6 would obtain reduced training time, and storage cost, while maintaining the true diversity of the datasets. As only the ND images are removed, and validation accuracy is retained. In scenarios where we want sacrifice some accuracy in trade of shortened training time, such as neural architecture search, or hyper-parameter optimization, we would suggest a threshold of 10 to obtain large dataset reduction (reduced by 77% for SVHN, with the accuracy dropped by 6%.

Related Works
Image hashing algorithms are proposed by compressing the original image into unique hash sequences [18]. In the domain of image deduplication, such image hashing algorithms yield identical hashes for images under slight transformation such as rotation, resizing, or color balancing, etc. Few applications are designed to meet above constraints based on image compression algorithms using Fourier transformation [28]. Venkatesan et al. [29] associate image hashes using coefficients of Discrete Wavelet Transform (DWT). In another approach, Discrete Cosine Transform (DCT) [1] is adopted to generate the hash sequence. Lin et al. [14] extend the work by associating invariant information between DCT coefficients. Layek et al. [12] leverage Locality Sensitive Hashing (LSH) to identify similarity for images in social media. Zhou et al. [31] propose applying LSH on image blocks to detect ND images for visual sensor networks. Rashid et al. [22] propose Set Partitioning in Hierarchical SIFT [16], SURF [2], ORB [23], feature extraction-based approaches. As future work, we would also like to test a more diverse set of image datasets such as MS-COCO [15] for object detection, and more CNN architectures.