{'fraction_non_alphanumeric': 0.03286335944299391, 'fraction_numerical': 0.012219321148825065, 'mean_word_length': 4.8291396103896105, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Transfer Learning based Feature-Weak-Relevant Method for Image Clustering

Image clustering is to group a set of images into disjoint clusters in a way that images in the same cluster are more similar to each other than to those in other clusters, which is an unsupervised or semi-supervised learning process. It is a crucial and challenging task in machine learning and computer vision. The performances of existing image clustering methods have close relations with features used for clustering, even if unsupervised coding based methods have improved the performances a lot. To reduce the effect of clustering features, we propose a feature-weak-relevant method for image clustering. The proposed method converts an unsupervised clustering process into an alternative iterative process of unsupervised learning and transfer learning. The clustering process firstly starts up from handcrafted features based image clustering to estimate an initial label for every image, and secondly use a proposed sampling strategy to choose images with reliable labels to feed a transfer-learning model to learn representative features that can be used for next round of unsupervised learning. In this manner, image clustering is iteratively optimized. What's more, the handcrafted features are used to boot up the clustering process, and just have a little effect on the final performance; therefore, the proposed method is feature-weak-relevant. Experimental results on six kinds of public available datasets show that the proposed method outperforms state of the art methods and depends less on the employed features at the same time.


Introduction
Cluster analysis [1] plays an important role in data mining and machine learning, whose mission is to partition objects into independent groups or clusters. Image clustering aims to classify images into categories based on image similarity, and it is valuable for future studies especially in machine learning. Desirable clustering results can ensure efficiency of further explorations. There are many classic clustering methods, such as K-means [18], GMM [19], Mean-shift [45], Spectral Clustering [46], DBSCAN [47], etc. Whatever kinds of clustering methods, the employed clustering features still have great effects on clustering results.
Two types of clustering features are always used for image clustering:  Handcrafted features According to prior knowledge of the sampled images, handcrafted features, such as color histogram [48], SIFT [49], HOG [34], Gabor [33], and LBP [50], can be applied for a quick image clustering. However, the clustering results can be unsatisfying when the feature space is indistinguishable for part of clusters. When implemented with different features, image clustering performs unstably because of its strong relevance to the employed features.  Unsupervised encoding based features Unsupervised Learning can be used for image encoding/decoding. The encoded codes are usually robust to noise and preserves information almost matches the original data. Therefore, the codes can be employed in image clustering. Unsupervised learning based clustering is mainly classified into two types: encoding based on linear models and encoding based on nonlinear models. The former type tends to map original data to a more distinguishable feature space by linear optimization. These methods can be well performed in data mapping, but linear models may be too simple to handle various image sets for clustering. For the latter method, deep neural networks (DNNs) based unsupervised learning techniques have been widely used for data preprocessing, especially DNNs based auto-encoding techniques. Though these techniques are good at data remapping and dimension reduction, there are no cooperative links between clustering and learning. That may lead to undesirable results at the same time [31]. Therefore, it is considered that mapping and clustering can be implemented jointly. The typical algorithms based on the 'joint idea' include JNKM [30], DEC [27] and DCN [31]. Actually two types of optimization objectives are considered as the minimum reconstruction errors and clustering losses. However, there exists a contradiction in the joint process: the minimum reconstruction errors guarantee well preserved information including interference information of images, which may lead to big clustering losses. At the same time, the joint process consumes too much time on DNN optimization, which limits further improvements in this field.
Compared with handcrafted features and unsupervised encoding based features, features learned by supervised learning are more representative and discriminative, but the process of acquiring features needs a large amount of labeled images which are not provided for image clustering. An alternative way is to generate clustering features using a Convolutional Neural Networks (CNN) [21] pre-trained with labeled images. However, this idea may not work out well when the labeled and unlabeled images are independently distributed in feature space. Fortunately, fine-tuning the pre-trained CNN [20], as one type of transfer learning methods, is a better way to overcome the shortcomings of direct features generation. At the same time, this process still needs supervised information such as labels.
Our idea is to use images with credible labels obtained by handcrafted features based clustering for booting the transfer learning process, and the features learned by transfer learning can be used for another round of clustering to get more images with credible labels which are used for next round of transfer learning. In this manner, more and more sampled images with credible labels are obtained, and image clustering can be iteratively optimized. An illustration of the idea is shown in Figure 1. To implement the idea, we propose a transfer learning based feature-weak-relevant method for image clustering. The proposed method aims to make image clustering depend less on handcrafted features by transforming an unsupervised learning process into an alternative iterative process of unsupervised  (2) where () t  is the mapping function. Typical linear-based () t  s include Principal Component Analysis (PCA) [9], Canonical Correlation Analysis (CCA) [10], Nonnegative Matrix Factorization (NMF) [11] and Sparse Representation (SR) [12]. Nonlinear-based () t  s include Stacked Auto Encoder (SAE) [13], Sparse Auto Encoder [14], Denoising Auto Encoder (DAE) [15] and Contractive Auto Encoder (CAE) [16]. As shown in (1) and (2), there are no mathematical connections between feature extraction and clustering, which makes the clustering results not satisfying enough and extracted features less representative. However, the mapping techniques are still popular among the researches on image clustering. Recently, instead of proposing the handcrafted dense/sparse feature models, in order to embed effective mathematical connections between features and clustering, researchers [27][28][29] have been studying on linear/nonlinear generative model from the latent space to the data domain while clustering.
One of the most impressive achievements is shown in [30].  define (1) as follows: is a parameter for balancing data fidelity and the latent cluster structure. 1 () r  and 2 () r  are pre-defined regularization parameters. In this way, () g  and () t  can be simplified into a trainable and question-related linear model.
For further study,  believe the data generating process is too complex to be precisely approximated based on a linear transformation, and the mapped features may not be representative enough. Therefore, in [31], they introduce Deep Neural Networks (DNN) into (3) as follows: ()  is the least-squared loss function for measuring the reconstruction error. () g  and () p  represent the encoding part and the decoding part respectively. It is an excellent idea to perform clustering with dimensionality reduction based on nonlinear SAE jointly for handcrafted features are no longer needed. SAE based (4) can automatically map from high-dimensional data to its latent space and the loss function of K-means clustering guarantees that the mapped feature is suitable for effective clustering.
However, for all encoding based () g  s or () t  s, the encoded F are redundant [17] because each i I consists of clustering relevant data i Ig and irrelevant data i Ib as shown in (5): (5) Hence, there exists a contradiction in (4): the minimum reconstruction error can guarantee F maintain all image information including interference features have negative effects on image clustering. It is an efficient way to embed () g  into clustering process as mentioned in the latest researches [27~31], and there exists improvements in feature simplification and clustering. In addition, SAE based clustering is working with high time and memory consumption in DNN training.

Formulations of the prototype
We hope images in the same cluster to be more similar to each other than to those in other clusters, and the cost function can be described as (6): )) However, equation (6) can be solved in an iterative form by assuming an initial feature extraction function 0 () g  , and the iterative form of equation (6) is : ) ( t g  is the feature function to be learned. The alternative iteration solution of equation (7) is as follows: are obtained by minimizing the first term of equation (7), i.e.: is the estimated labels of X , we can get *t L the clustering optimization.  t v is determined by a sampling function ()   whose goal is to choose reliable labelled samples with t M and t s as input, i.e.: ) ( t g  is learned by minimizing the 2nd and the third term of equation (7), which is to say: As long as labelled images are provided, equation (10) can be solved based on CNN classification models and this process is shown as follows: (11) tt  X (X X) denotes the reliable labelled samples at the tth round of iteration. Features that are more valid can be generated by computing values from any layers of the fine-tuned net, which can be described as follows: Alternatively, we can just use the trained model for updating *t L instead of (8) by prediction as (13) shows: As features are generated by Nets based on (12), the updated *t L will be more reliable. We can train models to be more effective using more images with credible *t L . However, it may lead to an over fitted CNN model when only a small amount of images is supplied for training. Commonly, we cannot ensure the credible labelled images are enough or not. Thus, there exists a risk generating unsatisfying models when solve (9) directly. In order to avoid the problems of divergence and overfitting caused by distribution of training data, the transfer learning method of fine-tuning a pre-trained classification model is adopted. And pre-trained structures include LeNet [22], AlexNet [23], VGG [24], GoogleNet [25], ResNet [26] and others.
Among the above existing structures, AlexNet is one of the most popular structures. Though AlexNet only consists of 5 convolutional layers and 3 fully connected layers, it performs well in the ILSVRC-2012 with a top-5 test error rate of 15.3%. Assuming the pre-trained AlexNet is ANets , model (11) can be then changed as follows: In conclusion, we need to initialize X first for booting transfer learning before the feature optimization and then implement the optimization by fine-tuning models and increasing the number of images that satisfy **t ii LL  . Therefore, we define the initialization part as the boot stage, and the initial features as the boot features. All other steps are called the main stage, and the final features can be more question-relevant. At last, model (6) can be solved by iteratively updating *t L based on transfer learning.

  Related Definitions
According to the definition of ()   , i v can be described as follows: (15) where () c  is a function for measuring cohesion between i I and j I with the same label *t i L . As i v is a binary label for grouping L , () c  can use the thresholding method for binarization as follows: (16) where () b c  represents inner distance evaluation models for images with same labels. According to clustering theories, there is a probability that images with features that densely distributed around the same center belong to the same class. The closer images are to the same center, the greater the above probability will be. The closer images X with 1 N  v1 are called Cluster Center Neighboring sets (CCNs) and the rest with are called Cluster Center Distant sets (CCDs). 12 , c  can be defined as (17): (17) There are many alternatives for distance measuring in (17), such as Chess Board Distance, City Block Distance, etc.
In (16), the threshold Δ is important for binarization. A proper threshold should be given based on the clustering features before optimization, and it varies with different types of handcrafted feature. Therefore, a constant Δ can be initialized easily, but it is not effective for all features. To avoid the issue, we define Δ as a relative ratio that can be also initialized simply. Instead of referring to the results of () b c  directly, we sort the results on same labels in descending order as (18) shows.
is a vector containing the sorted distances. We set Δ is considered as a belief ratio indicating there are Δ% CCNs in every cluster. The relative threshold is more robust to variation of the employed handcrafted clustering features.

Optimization Procedure in Main Stage
In section 3.1, we convert (6) into an iterative optimization model that can be solved by iteratively updating *t L . In order to make the optimization implementable, we need to add following three constraints (assuming k is the iteration counter, k   Z ):  CCNs and CCDs can never be the same until the optimization is done, and .  The CCNs cannot be modified after confirmation, and 11 11

Algorithm Description
Based on constrains in section 3.3 and the implementation analysis, we can update

Experiments
In this section, various real-world data sets, different types of initial feature sets and clustering methods are applied to the proposed method for effectiveness validation, respectively. In sections 4.1~4.3, we provide detailed information of implementing relevant experiments, including baseline methods, evaluation models and public image sets for clustering. In section 4.4 different comparative experimental results are respectively analyzed. Then we investigate the sensitiveness of the parameter Δ and δ in section 4.5. Meanwhile the experiments on time consumption are analyzed in section 4.6. All fine-tuning experiments are implemented with Caffe [32] and the detailed information of the prototxt file related to fine-tuning is available at https://github.com/DukeDong/Transfer-Learning-Based-Image-Clustering-.git.

Baseline Methods
Clustering features and methods have great effect on the performance of an image clustering algorithm, so we choose the following features, methods and their combinations for evaluations.  [33]. This kind of feature shows the information in spatial and frequency domain.  HOG Features: histogram of gradient [34]. HOG represents edges of objects in images, and it is robust to geometrical and optical deformation.  RP + PCA: feature vectors generated by PCA. The combination is still applied in face recognition [35].

Clustering methods:
 K-means [18]: the classic cluster method.  Gaussian Mixture Model (GMM) [19]: the cluster method based on a latent variable model.  Deep Clustering Network (DCN) [31]: a new method that can jointly optimize dimension reduction and clustering. It is accomplished via learning a deep neural network to approximate any nonlinear function.
There are nine kinds of combinations between different features and methods for image clustering including one method based on DCN, which are shown in table 1.

Evaluation Models
There are mainly two kinds of models for performance evaluation of a clustering method, which are external index and internal index. The applied models with external index include Jaccard Coefficient (JC) [36], Fowlkes and Mallows Index (FMI) [37], Normalized Mutual Information (NMI), Clustering Accuracy (ACC) [38] and Adjusted Rand Index (ARI) [39]. As to internal index, we employ Davies-Bouldin Index (DBI) [40]. Comparing DBI directly is meaningless when different features are employed for clustering. Therefore, we calculate final DBI by normalizing DBI gained in the boot stage. In this way, the normalized DBI implies a reduction of the initial DBI.
Besides, one of our goals is to weaken the effect of initial features. Another evaluation factor is required to examine how much effect the proposed method can weaken the initial features. Assuming clustering results with different initial features for a clustering method are available, the evaluation factor  can be defined as (19): (19) where i  is the index difference rate,

Data Sets
Image clustering methods need to handle many different kinds of objects. In order to evaluate the generalization of the proposed method, experiments are conducted on six typical public image sets:  MNIST [22]: MNIST dataset has 70,000 data samples of handwritten digit ranging from 0 to 9.  Images in Caltech Categories are more complicated than other applied data sets, and different image classes require different features.

Performance on Caltech Categories
Though Caltech Categories only contains six categories, it requires image preprocessing to achieve desirable clustering results. Image preprocessing is essential for feature selection in prior researches. However, in order to make the process more challengeable, we employ #2~#4 and #7 directly without any preprocessing related analysis. Clustering with the feature RP directly is meaningless, so #1 and #5 are omitted. In fact, it is effective enough that GMM performs image clustering with only one type of handcrafted features for comparison. For GMM, we only list out the results of #7. In the experiments, Δ is set to 0.2, and  is set to 0.1, which is applied in sections 4.4.2~4.4.6. In addition, the cluster number is set to six for K-means and GMM. Table 2 shows the five evaluation indexes. Table 3 shows the index difference rates with different features. The results show that the proposed method performs much better than other methods and meanwhile it weakens the effects of initial features.

Performance on Caltech 101
Compared with Caltech Categories, Caltech 101 dataset contains more various categories, but each category has fewer samples, sample number of which is round ninety on average. Therefore, Caltech 101 can be used to validate the robustness to the variation of image numbers and cluster numbers.
We randomly choose 20 categories from Caltech 101 in this part, and repeat the random process for more than 3 times to guarantee the effectiveness of our experiments. Accordingly, the cluster number is set to 20 for K-means and GMM. Here, the category number of 20 is larger than common maximum cluster number for clustering evaluation of 2, 5, 10, etc. More clusters mean more difficult clustering mission. However, when we conduct image clustering with bigger clustering numbers, we suffer from high time consumptions of K-means and GMM. Therefore, cluster number of 20 is a rough but proper for clustering performance evaluation.  Table 4 shows the detailed evaluation indexes. Table 5 shows the index difference rates with different features.
As the experimental results show, the proposed method not only surpasses other methods, but also with the better external index values. At the same time, the proposed method weakens the effect of initial features. However, the results of DBI are unsatisfying, which means the learned features is not densely distributed in feature space and further studies may improve DBI and clustering results. categories into different clusters. If the results turn out to be satisfying, the corresponding transferlearned features are believed to be desirable and independent from the original AlexNet. Stanford Dogs dataset has been built using images and annotations from ImageNet for fine-grained image categorization. The annotations include class labels and bounding boxes. It helps a lot when an image clustering algorithm is provided with bounding boxes, but the bounding boxes are hardly available during unsupervised image clustering. In order to make the experiments more reliable, the box information is omitted in this section. Compared with the Caltech datasets, Stanford Dogs Dataset contains categories representing clusters in a more detailed way, and the average number of sample images in each breed is approximately 172. We randomly choose 20 breeds from the database this time, and repeat the random process for more than 3 times to ensure the effectiveness of the experiments. The cluster number is set to 20 for K-means and GMM. Table 6 shows the detailed clustering evaluation indexes. Table 7 shows the index difference rates with different features. Similar results to those obtained in section 4.4.2, the results of DBI evaluated with the proposed method are turned out to be even larger. As the learned features can be further explored, the clustering performance is highly possible to be improved. However, the detailed external indexes still imply that the proposed method highly improves the clustering results and weakens the effects of initial features.

Performance on FERET
The FERET database is used for face recognition system evaluation. As the collection is progressed in a standard way, the chosen images contain just a little background information and many various face photos with different postures.
Different from the previous datasets mentioned above, FERET is classified individually, which means FERET clustering is a process of unsupervised face recognition. Face image clustering is much closer to the leaf nodes than clustering methods based on Caltech categories and dog breeds. It is a traditional way to build up suitable features when the images can be clearly noticed. In order to make the experiments more reliable, the box information is omitted in this section as well. In this part, we randomly choose 20 individuals from FERET, and repeat the random process for more than 3 times to make sure the effectiveness of the experiments. The cluster number F is directly set to 20 for K-means and GMM.  Table 8 shows the detailed clustering evaluation indexes. Table 9 shows the index difference rates with different features. Similar results to those reported from Sections 4.4.3 and 4.4.4, the results of the proposed method in DBI are larger than the corresponding ones obtained from initial methods. All the learned features are with the same dimension of 4096, which is fixed and probably redundant. Meanwhile, the data shows the proposed method performs better and weakens the effect of initial features.

Performance on CASIA-WebFace
Images in CASIA-WebFace are collected from the Internet, which is different from image collections in FERET, therefore the CASIA-WebFace images normally carry much more background information than those in FERET. That means it is more challenging for image clustering with CASIA-WebFace.
We randomly choose 20 individuals from CASIA-WebFace in the same way, and repeat the random process for more than 3 times. The cluster number is directly set to 20 for K-means and GMM.  Table 10 shows the detailed clustering evaluation indexes. Table 11 shows the index difference rates with different features. Although CASIA supplies with more complicated face images, the results of the proposed method can be still achieved higher in external indexes and lower in internal indexes with less exceptions.

Performance on MNIST
Different from all the other datasets applied in sections 4.4.1~4.4.5, MNIST is a dataset of handwritten digit images. The images in MNIST are both simple and various. There are no related background information or background noise in each digit image. Thus MNIST is an ideal dataset for image classification. Meanwhile it is considered as a pre-processed dataset for image clustering, and every type of classical features can achieve good results including RP. For all images with a size of 28×28 pixels, the clustering based on DCN has proved to be well implementable.
Here we set Δ to 0.2 and  to 0.1, and the cluster number is set to 10 for K-means and GMM.
The results of performance evaluation with the proposed method except DCN-based method are displayed in table 12. Table 13 shows the index difference rates with different features. Table 12 shows, the results generated by transfer learning based clustering are improved. Although the initial features have effects on clustering, the proposed method can weaken the effects a lots. Of course, better initial features guarantee to achieve labels that are more correct and much better results.
DCN uses a forward network with 4 layers and sets the numbers of neurons to be 500, 500, 2,000, and 10. The reproducible codes remain mostly unchanged from https://github.com/boyangumn/DCN. However, we failed in reproducing clustering results based on #9. Therefore, we use their results directly and simulate correct labels according to its ACC performance in raw MNIST. Meanwhile when we change one cluster image with a size of 28×28 pixels to another one with a size of 256×256 pixels, DCN faces the curse of dimensionality. And the configurations for the depth of layers and number for each layer are unavailable. We use K-means instead of DCN in optimization part. It is pity that DCN can be comparable only in the experiments related to the image clustering based on MNIST. According to DCN's ACC index of raw MNIST, there are 83% correct labels in every cluster. Thus, we randomly choose 83% images from a same class in MNIST, and the rest of unchosen images are labeled as wrong labels. In this way, the initial evaluation indexes are worse than the actual results produced by DCN. Moreover, we repeat the simulation for more than three times to guarantee the effectiveness of the experiments.
The actual indexes, simulated indexes and indexes gained with the proposed method are shown in table 14. Although the simulated data are not perfectly reproduced, our method still improves the cluster performance.

4.5
Sensitivity of Δ and  Δ , theoretically represents the confidence values of clustering results. If the value of Δ is larger, then the initial clustering is supposed to perform more desirably. The progress can possess more time consumption without  . On the contrary,  may lead to local results to be unsatisfying. Thus, it is necessary to examine the effect of these two parameters on the proposed method.
We just employ ACC as the model for performance evaluation in this part and perform the experiments with sampled Caltech 101 and Stanford Dogs. The results are shown in Figure 2.
 can fasten the optimization with a relatively small number of iterations. Figure 3 shows counts of iterations with different  and fixed Δ=0.2 . As shown in Figure 2, the configurations of Δ and  have effects on the results. Δ is relatively more important to the result because a larger Δ can introduce more noisy labels in training. A proper Δ can ensure effectiveness of learned features. When little prior knowledge is supplied, it is a suitable choice to set Δ with values close to 1/ K , which is the ACC of a random clustering. A larger δ may make the proposed method stop faster, but accordingly  is suggested to set with values close to 0.1.
In this case, it can hardly affect the results and decrease the time consumption as Figure 2 shows.

Time consumption
In this section, we list the employed hardware equipment and software in table 15. Actually many factors can influence the time consumptions on implementing experiments, including the number of images in different datasets, images with different resolutions, and configuration parameters. In table 16, we show the maximum statistical time consumption on (usually goes with GMM) experiments with the employed datasets and the relative rough data.
The time consumptions are acceptable but not satisfying enough. Even though the number of images in FERET is set to 140, transfer learning by fine-tuning a pre-trained model still spends around half an hour. In addition, as shown in table 16, the implemented codes are combinations of matlab, python and C++ (Caffe), which means there are extra more time consumptions on data conversion. Maybe we will later try to find out a faster way on image clustering based on transfer learning.