{'fraction_non_alphanumeric': 0.02556463625749829, 'fraction_numerical': 0.010386891581786706, 'mean_word_length': 5.26008064516129, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enhancement of the Heuristic Optimization Based on Extended Space Forests using Classifier Ensembles

Extended space forests are a matter of common knowledge for ensuring improvements on classification problems. They provide richer feature space and present better performance than the original feature space-based forests. Most of the contemporary studies employs original features as well as various combinations of them as input vectors for extended space forest approach. In this study, we seek to boost the performance of classifier ensembles by integrating them with heuristic optimization-based features. The contributions of this paper are fivefold. First, richer feature space is developed by using random combinations of input vectors and features picked out with ant colony optimization method which have high importance and not have been associated before. Second, we propose widely used classification algorithm which is utilized baseline classifier. Third, three ensemble strategies, namely bagging, random subspace, and random forests are proposed to ensure diversity. Fourth, a wide range of comparative experiments are conducted on widely used biomedicine datasets gathered from the University of California Irvine (UCI) machine learning repository to contribute to the advancement of proposed study. Finally, extended space forest approach with the proposed technique turns out remarkable experimental results compared to the original version and various extended versions of recent state-of-art studies.


Introduction
Ensemble learning is a very popular research area in the literature and has also known as committees of learners, mixture of experts, ensemble of classifiers, ensemble algorithms [1,4,14,16,35]. The idea behind of this principle is to make use of more than one classifier. Thus, the classification task is performed more robustly and accurately [10,24,26,27,28]. Supervised machine learning techniques such as naïve Bayes classifiers, decision trees, Support Vector Machines (SVM), Artificial Neural Networks (ANN), k-Nearest Neighborhood (k-NN) are commonly used for the ensemble strategies. Especially, decision tree is more widely implemented in the literature for ensemble classifiers compared to the other classification methods [1,2,3,4,5,11]. Utilization of more than one decision tree emerges decision forests for the classifier ensembles.
As much as the selection of classifier, the individual success and diversity of base learners are also determinative factors of the ensemble success. As the diversity of base learner increases, the classification success of system becomes better. The usage of different or same base learners is requisite in order to provide diversity. Diversity is maintained with several conventional ensemble algorithms such as bagging, random subspaces, random forests, and rotation forest for the same base learners. For different base learners, it is already achieved by blending different learning algorithms with various decision making techniques such as majority voting, stacking, cascading. In this work, we focus on the same base learners to provide diversity is called homogeneous classifier ensembles.
In this paper, we propose to contribute to the classification performance of ensemble system using heurictic optimization based technique namely, Ant Colony Optimization (ACO) for biomedicine datasets. For this purpose, extended space forest employs the association of randomly chosen features and ACO based significant features to construct extended feature space. Then, decision tree construction is carried out on training phase according to the ensemble algorithms namely, bagging, random subspace, and random forest. For demonstrating the contribution of proposed technique, we conduct experiments on datasets available from the University of California Irvine (UCI) machine learning repository [8].
The rest of the paper is organized as follows: section 2 gives related researches on the use of ensemble systems and ant colony optimization. In section 3, we give proposed feature extension technique employed in the experiments. Experiment setup and results are demonstrated in sections 4 and 5. Section 6 concludes the paper with a discussion and conclusions.

Related Work
Ensemble learning is the collection of methods that builds a set of classifiers and combines their classification predictions by using majority voting [1,4,10,13,23,30]. Previous studies [20,26,29,30,32] have indicated that the final decision of ensemble system is more accurate and robust than any of the individual classifiers in the ensemble. In an interesting study [34] on ensemble of feature sets and classification algorithms, authors focus on the effectiveness of ensemble methods for sentiment classification. Experimental results show that the usage of ensemble methods for both combining different feature sets and classification algorithms is efficient to boost classification performance. Another important study [17] on ensemble learning proposes the novel SVM technique namely, the Ensemble of SVM (EnSVM), to excel the classification success of SVM on imbalanced data. The extensive experiments demonstrate that their proposed technique is competitive, effective and superior in proportion to several methods with various data sampling techniques. The other study also [25] proposes to investigate the effectiveness of the ensemble classifiers for offline cursive character recognition. Their experiment results point out that the accuracy of recognition success using ensemble algorithms is better than the traditional recognition accuracies for offline cursive character recognition.
There are also several studies on classifier ensembles with extended space. The influential study [4] proposes the extended feature space by choosing new features randomly and adding them to original feature space. They observe that all extended versions outperform original versions for all ensemble algorithms. To get higher classification performance of ensemble system, they suggest utilizing the extended space methods. The recent studies [1,2] on extended space decision trees propose to increase the ensemble accuracy by suggesting another approach. Instead of randomly producing, new features with high classification capacity are generated by computing the gain ratio of each different candidate features. Thus, they combine newly generated features and existing features in order to extend feature space. Authors conclude that the extended space forest is an effective method to increase prediction accuracy but it can be improved by using significant features instead of selecting randomly. Another recent study [32] investigates the effectiveness of enhanced Random Subspace Method Based on Part-of-Speech Technique (POS-RS), for sentiment categorization field. They conclude that POS-RS is preferable method to excel the success of classification and applied to the other text classification problems.

Proposed Technique and Ensemble Strategies
As noted in the previous works [1,2,4] the enrichment of feature space ensures significant contribution to the classification performance. The studies so far on extended space forests utilize either randomly chosen features [4] or the specific feature selection method such as gain ratio [1,2] to determine new candidate features to be consolidated to the original feature space. In the proposed approach on extended space forests, we take the approach one step further in extended space forests by employing heuristic optimization technique namely, ACO as a feature selection method in addition to the random combinations of input vectors. The ant colony optimization is an optimization technique that can be also employed for feature selection on various domains. It is based on finding the shortest paths from the nest to food source by means of pheromone trails, which is an odorous substance and is excreted by ants. Therefore, the deposition of pheromone is the fundamental factor in order to discover the shortest paths over a certain period of time. The way excreted pheromone is used by more ants and pheromone trails probabilistically enforce to choose the previously marked path for each isolated ant. On less preferred paths, pheromone evaporates over time and the shortest path is discovered by means of the higher ratio of ant traversals. For this reason, there is a transition probabilistic rule for each ant to determine the probability of being selected corresponding path. Hence, ACO technique is attractive for feature selection process that can direct search to optimal subset every time [9]. Easy implementation and superior performance of ACO encourages us to enrich feature space. When 100 base learners are thought to be used for ensemble strategy, it is expected to use random features for each base learner (for each isolated ant) at first. Furthermore, the pheromone intensity is updated on the route drawn for each ant to avoid stagnation and then the isolated ants can choose different routes (features). Therefore, the extension of feature space can be provided with different features for each base learner. The probabilistic transition rule, expressing the probability of an ant at feature i choosing to travel to feature j at time t: where k is the number of ants, ηij is the heuristic desirability of selecting feature j when at feature i, is the set of ant k's unvisited features, and τij (t) is the amount of virtual pheromone on edge (i, j), α provides global information and determine the relative importance of the pheromone value, β is the heuristic information and present local information. Producing a (1) number of k ants is the first step for ACO feature selection process. In this study, the number of ants is set to the number of features within dataset. Thus, each ant begins with one random feature and they travel edges probabilistically until stopping gauge is fulfilled. The subsets are congregated and then evaluated. Once the algorithm has performed a certain number of times or an optimal subset has attained, the overall feature selection process terminates by obtaining the best feature output. If neither condition holds, it is inevitable to update the intensity of pheromone, then new ants are produced and the feature selection process reiterates once more. Ultimately, the extended feature space comprises original, randomly chosen features and significant ones selected with ACO. In addition to the d number of original features, the d number of space extension parameter is adjusted to extend feature space due to its superior performance as stated in [4]. While the first half of new features is composed from significant features chosen with ACO, the remaining half is generated as the randomly combined pairs of the original features. After constructing the enriched feature space, the extended space forest is evaluated with the well-known classification algorithm, namely decision tree by using three types of ensemble algorithms such as bagging, random subspaces, and random forests. Figure 1 illustrates the process of extended space forest with proposed technique. Ensemble algorithms used in this work are briefly mentioned. Bagging [4,6,12,18,21,31,32,33] generates new bootstrap samples utilizing substitution from the original dataset. Then, training is done by constituting on each of these samples and associated them with majority voting ensemble strategy. Random Subspace [4,11,12,15,19,21,22,31,32] exploits fairly simple randomness approach for the feature selection. Training is done with a subset of the original feature space instead of including all features for each base learner in the ensemble. Then, the classifier is constructed on different feature subsets illustrated randomly from the original feature set and associated by applying the majority voting. Random Forest [2,4,7] combines two approaches namely, Bagging and Random Subspace algorithms. The proposed approach is also described in details below. In proposed approach, the parameters are given as: E={xp, yp}p=1…N =[X Y] where X is an N*d matrix including the training set and Y is an N dimensional column vector covering the class labels. N is the number of training samples, d is the number of features, T is the number of base learners, BLi is the base learner, Ei is the enhanced training set for BLi, Ri consists of feature pair indices used in generation of Ei, EA is an ensemble algorithm. At initialization step, ensemble size T, the base learner model BLi, and the ensemble algorithm EA are chosen.

Experiment Setup
We utilize datasets with different sizes and properties that are available from the UCI Machine Learning Repository [8] in experiments. Characteristics of the datasets are presented in Table 1 including the number of features (|F|) and the number of classes (|C|). Experiments are carried out by modifying the training set levels and utilizing 10%, 30%, 50%, and 80% percentages as a training data. The accuracy percentage levels are abbreviated with "ts" affix to head a commotion off. The algorithms are launched at each training set levels by partitioning 10 parts randomly and stratified sampling is exploited at this step. Statistical significance tests are also employed in several places by evaluating Student's t-test which is beneficial as the closeness of accuracy results of different techniques is observed. Significance level is set to 0.05 and the difference is accounted as statistically significant when the association of probability and Student's t-Test is lower. The number of base learners is adjusted to 100 as represented in [2,4]. As we mentioned before feature extension parameter is set to d number of features for all datasets for comparing experiment results with impressive work [4]. To combine the decisions of base learners, majority voting is employed for all ensembles. Moreover, it is necessary to specify some parameters for ACO feature selection process. Firstly, the number of ants is equal to the number of features for each dataset. Because of this, the number of ants varies according to the dataset. Then, the algorithm has carried out a certain number of times is the same as the number of base learners, i.e., 100 times. After the algorithm has executed 100 times, the pheromone density is updated and a new set of ants are composed and the process iterates once more. The initial pheromone density of each feature is set to 1 at first, two important information, local and global, about the traversal of ants is determined with the parameters α and β. The choice of α, β is specified experimentally and set to 1 and 0.1, respectively. The pheromone trail evaporation coefficient (ρ=0.2) is a parameter to update pheromone trails and located in the range between 0 and 1.

Experiment Results
Ensemble Accuracy (EA) is utilized as an evaluation metric to demonstrate the contribution of our work. Abbreviations are employed for the ensemble algorithms and feature selection techniques as follows: BG: Bagging, RS: Random subspaces, RF: Random forests, X0: Original version of dataset for X ensemble algorithm, XRD: Extended space forest by adding randomly selected features for X ensemble algorithm, XACO + RD: Extended space forest by adding randomly chosen and ACO based features for X ensemble algorithm.
The classification success is ordered as RFACO + RD > RSACO + RD>BGACO + RD > RFRD > RSRD > BGRD>RFO > RSO >BGO at ts80 in terms of ensemble accuracies as seen in Table 2. Except ts30, the best classification performance is performed by RFACO + RD at all training set sizes. Thus, it can be asserted that the enhanced space forests based upon ACO and random features contribute to the classification performance significantly for 36 datasets. Except ts30, the success order of original versions of ensemble algorithms is RF > RS > BG. At smaller training set levels like ts30, the performance order of original versions of ensemble algorithms is different but not enough to claim statistically considerable because of the closeness of accuracy results. Moreover, RFACO+RD outperforms others at all training set percentages except ts30 level. At ts30, RSACO+RD is competitive and surpasses other techniques by at 1%. The combination of random subspace as an ensemble algorithm and ACO based and randomly selected features as a feature selection technique yields by far the highest accuracies at this training set level. At the last of the Table 2, average accuracy results are presented. Average accuracy results for bagging algorithm demonstrate that the extended space forests by adding randomly selected and ACO based features is the winner with 87.9% accuracy value at ts80 compared to others. The success order of original and extended versions of bagging algorithm is given as follows: BGACO+RD>BGRD>BGO. Similarly, the extended version with ACO+Randomly selected features (RD) outperforms others for the random subspace and the random forest algorithms with 88.1% and 88.4% accuracy results, respectively. Like bagging algorithm, the performance order of random subspace and random forest is the same. Hence, our proposed method based extended space forest (ACO+RD) is the best model to enhance the classification accuracy for each ensemble algorithm in terms of average accuracy results.
For the original space forests, random forest is the best ensemble algorithm with 87.0% classification success and followed by random subspace with 86.9% and bagging with 86.2% accuracy result, respectively. The classification performance of ACO+RD based extended space forests is ordered in a similar way as: RF>RS>BG and the classification performance is consistent with the state-of-art results [4]. This order is also valid for the extended space forests with randomly chosen features but the classification accuracies are different from each other in that 87.8% RF, 87.7% RS, 87.3% BG. It is significant to perceive that the classification results of random forest and random subspace algorithms are close to each other but yet, random forest algorithm generally surpasses the others at ts80 by evaluating Table 2. As a feature selection technique the consolidation of random and ACO based features is an optimal in order to enrich feature space. As we pointed out above, if we compare all versions of the space forests according to the ensemble algorithms, we can attain the classification success order as: RFACO+RD > RS ACO+RD>BG ACO+RD >RFRD>RSRD >BGRD>RFO > RSO>BGO. In this study, we try to acquire better classification performance compared to the previous studies [1,2,4] and implement training procedure on the extended feature space with an appropriate feature selection model and then associate with various ensemble algorithms.
Experiment results exhibit that the combination of random and ACO based features and random forest as an ensemble algorithm has an exceptional classification performance.
Unlike smaller training set percentage levels, random forest algorithm with ACO+RD reaches the maximum value at ts80. All versions of random subspace algorithm have the following best classification performances. So long as training set percentages increase, the successes of all enhanced space forest versions also rise up and vice versa. Table 3. Comparison between pairs of algorithms: "win (significant win)/ loss (significant loss)" row vs. column. Original versions of all ensemble algorithms present the lowest classification accuracies at higher training set percentages. At these training set levels, the choice of original versions of ensemble algorithms will not be a good preference for the classification problems. The following consequences can be drawn from Table 3: BGACO+RD has higher accuracy than BGO over 28 datasets out of 36, and has 15 significant wins. RSACO+RD has higher accuracy than BGACO+RD over 21 datasets out of 36, and has 11 significant wins. RFO has higher accuracy than RSO over 22 datasets out of 36, and 4 significant wins. RFRD has higher accuracy than RSRD over 19 datasets out of 36, and 5 significant wins.

Discussion and Conclusions
The superiority of ensemble systems is a widely accepted assumption in machine learning domain as mentioned before. Owing to this approach, it is recommended to produce more accurate and robust models. In this work, we propose to investigate the contribution of enhanced space forests to the classification performance by utilizing ensemble algorithms. For this purpose, we take the concept one step further in extended space forests by employing ACO as a feature selection technique which have not been combined before. Moreover, this is the first research for the extended space forests with classifier ensembles in terms of utilizing heuristic optimization technique, ant colony optimization, as a feature selection technique. Features chosen with ACO and random combinations of input vectors are blended with the original features to constitute a new extended feature space. Then, the enriched feature space is carried out on three popular ensemble algorithms (Bagging, Random Subspaces, and Random Forest) by utilizing decision tree as a base learner. Finally, the extended space forests developed by the proposed method ensure noteworthy enhancement to the classification performance in comparison to the original version and various extended versions of recent state-of-art studies. Considering the overall classification performances, space forests with the original feature space have the lowest accuracy results at all training set levels and this is an indicator that the space forests having original feature space tend to development.
As well as the classification success of enhanced space forests, execution time analysis is also appraised in terms of testing and training times. More training time is necessitated for the enhanced space forests in proportion to the original ones owing to covering more features and directly proportional to the search time of the features, vice versa. The complexity of base learners which drops a hint about the testing time is proportional to the number of nodes in a tree. Hence, the most complex base learners are constituted by random forest algorithm due to having the biggest trees. It is substantial to emphasize that the enhanced space forests are entailed less testing time because of having smaller trees compared to the original space forests.
It is considerable to compare experiment results with the state-of-art studies [2,4] on extended space forests to demonstrate the contribution of the proposed technique. Authors employ 36 datasets from the UCI repository [8] and all of them are common with ours. The classification success of randomly enhanced space forests which is proposed by [4] is consistent with our experiment results at ts50 in terms of ensemble algorithms. The classification performance of ensemble algorithms is ordered as: RF>RS>BG. Furthermore, the classification success of the proposed technique for bagging algorithm (86.1%) outperforms experiment results of their study (85.3%) with approximately 1% improvement. When it is considered on the random forest algorithm, the proposed technique turns out well with the accuracy rate of 86.5% whereas the study of Amasyalı and Ersoy [4] provides accuracy rate with 85.9%. Though the random subspace algorithm of both studies presents close experiment results, the proposed technique of ours boosts the classification performance of whole system compared to the study [4].
Another work [2] also strives to demonstrate the superior classification performance of enhanced space forest by using gain ratio technique for only random forest ensemble algorithm. They make use of ten different datasets from UCI repository and succeed 86.4% classification performance. Finally, the study is concluded that the great contribution of enhanced space forests to the classification success cannot be ignored. The slightest difference between ours (86.5%) and theirs (86.4%) can be arisen from the number of features, the number of datasets, differences in experimental settings and the ambiguity of the training set levels. Thus, it is noteworthy to specify that the combination of the proposed technique and random forest algorithm (RFACO + RD) predominantly surpasses state-of-art studies.
To sum up, the enhanced space forest approach advances the classification success of system compared to the original versions. over and above, it is observed that the enhancement of extended space forest with classifier ensembles using ant colony optimization as a feature selection technique exhibits better classification performance in proportion to the other enhanced space forest techniques. In future, we plan to carry out different base learners to the classification problems.