{'fraction_non_alphanumeric': 0.03549300761196672, 'fraction_numerical': 0.007302177376526819, 'mean_word_length': 4.546637211585665, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Editing Training Sets from Imbalanced Data Using Fuzzy-Rough Sets

. In this research, we study several instance selection methods based on rough set theory and propose an approach able to deal with inconsistency caused by noise and imbalanced data. Recent attention has focused on the signiﬁcant results obtained in selecting instances from noisy data using fuzzy-rough sets. For imbalanced data, fuzzy-rough sets approach is also applied before and after using balancing methods in order to improve classiﬁcation performance. In this study, we propose an approach that uses diﬀerent criteria for minority and majority classes in fuzzy-rough instance selection. It thus eliminates the step of using balancing techniques employed in controversial approach. We also carry out some experiments, measure classiﬁcation performance and make com-parisons with other methods.


Introduction
Rough set model is a machine learning technique for acquiring knowledge from datasets. Rough set theory was first introduced by Pawlak [19,20] in the early 1980s. It is a mathematical approach designed to deal with the ambiguity, vagueness and uncertainty that exist in a database. In knowledge discovery, rough set theory constitutes a sound basis by offering mathematical tools to discover patterns hidden in data. It can, according to [25], be used for feature selection, data reduction, decision rule generation, and pattern extraction.
Recently, rough sets have been implemented for instance selection [3,12,27]. The main ideas of these approaches are extracting fuzzy memberships of positive regions and choosing the instances which have large membership degrees for training phases. However, it is believed that these criteria do not overcome all problems in relation to inconsistency issues.
One issue that may cause an inconsistency problem is imbalanced data [14]. A dataset is imbalanced when the numbers of instances in some classes are much larger than in others. Such classes are called majority classes. The classes with small cardinality are referred to as minority classes.
In fact, some researchers are trying to use the advantages of rough sets to enhance classification performance when a balancing method is employed [22,23,28]. In these approaches, fuzzy-rough sets are first used to remove low quality instances. Then, a well-known balancing technique called "Synthetic Minority Oversampling Technique (SMOTE)" [4] is employed to form a candidate set. Finally fuzzy-rough sets are used again to select quality instances from the candidate sets. Obviously, such methods need considerable computation time to edit and select instances.
This research will introduce an approach that uses fuzzy-rough sets to balance and select quality instances from imbalanced datasets. In this work, different thresholds for majority and minority classes are used, thus, more items from minority classes can be selected. The experiment results show considerable improvement in classification performance and the advantage with respect to other methods in the literature.
The paper is structured as follows: Section 2 reviews the original rough set theory and an extension called fuzzy-rough set. In section 3, some rough set approaches for dealing with inconsistency are discussed along with their issues. The new algorithms and experiments are introduced in Section 4. Finally, we present our conclusions in Section 5.

Information Systems and Equivalence Relation
An information system is represented as a data table. Each row of this table represents an instance of an object such as people, things, etc. Information about every object is described by object attribute (feature) values.
An information system in the rough sets study is formally defined as a pair I = (U, A), where U is a non-empty finite set of objects called the universe and A is a non-empty finite set of attributes such that f a : U → V a for every a ∈ A [19,20]. The non-empty discrete value set V a is called the domain of a. The original rough set theory deals with complete information systems in which ∀x ∈ U , a ∈ A, f a (x) is a precise value.
Any information system taking the form I = (U, A ∪ {d}) is called a decision table where d / ∈ A is called a decision (or label) and elements of A are called conditions. Let V d = {d 1 , ..., d k } denote the value set of the decision attribute, decision d then determines a set of partitions Set C i is called the i -th decision class or concept on U . We assume that every object in U has a certain decision value in V d .
Formally, in complete information systems, relation EQU P (x, y), P ⊆ A, denotes a binary relation between objects that are equivalent in terms of values of attributes in P [19]. The equivalence relation is reflexive, symmetric, and transitive. Let E P (x) = {y ∈ U |EQU P (y, x)} be the set of all objects that are equivalent to x by P , which is then called an equivalence class. The family of all equivalence classes (or partitions) on U based on an equivalence relation is referred to as a category and is denoted by U/EQU P .
From equivalence classes, Pawlak [19,20] defined an approximation space that contains lower and upper approximations denoted by apprX and apprX, respectively, of set X ⊆ U as follows: In decision table I = (U, A ∪ {d}), the positive region P OS P (X) of set X in terms of attribute set P ⊆ A is defined as: Others region definitions such as negative, boundary and rough set properties can be obtained from the original rough set theory research [19,20].
Apart from using equivalence relations to define a rough set in complete information systems, there are also numerous studies [10,13,16,17,18] that deal with incomplete or imperfect information systems in which data are not described by precise and crisp values.

Fuzzy Rough Set
A classical (crisp) set is normally defined as a collection of elements x ∈ X that can be finite, countable or over countable. Each single element can either belong to or not belong to a set X ⊆ X. For a fuzzy set, a characteristic function allows various degrees of membership for elements of a given set. Then a fuzzy set X in X is a set of ordered pairs [31]: where µ X (x) ∈ [0, 1] is called the membership function or grade of membership (also the degree of compatibility or degree of truth) of x in X . In crisp sets, for two special sets ∅ and U the approximations are simply defined as µ appr R U (x) = 1 and µ appr R ∅ (x) = 0. Based on the two equivalent definitions, lower and upper approximations may be interpreted as follows: An element x belongs to the lower approximation appr R X if all elements equivalent to x belong to X. In other words, x belongs to the lower approximation of X if any element not in X is not equivalent to x, namely, µ R (x, y) = 0. Likewise, x belongs to the upper approximation of X if µ R (x, y) = 1. Now the notion of rough sets in crisp sets is extended to included fuzzy sets. Let µ X and µ R denote the membership functions of set X and of the set {(x, y) ∈ U × U |R(x, y)}, respectively. The fuzzy approximation space [21] of fuzzy set X on X in terms of fuzzy relation R(x, y) can be defined as follows: where I and T are fuzzy implicators 1 and triangular norms (t-norm) 2 , respectively. From the above equations, it is noticed that membership of lower and upper approximation for one instance strongly depend on only one other instance since they are defined by minimum and maximum. To soften this definition, Cornelis et. al. [5] suggest a fuzzy-rough set definition using order weight averaging (OWA) aggregation [29]. An OWA operator F W is a mapping F : R n → R using weight vector W =< w 1 , ..., w n > such that where b i is the i th largest of the values in {a 1 , · · · , a n }.
An OWA operator is bounded by the minimum and maximum of vector W . It thus can soften minimum and maximum operators. Suppose that there are two vectors W min =< w min 1 , · · · , w min n > and W max =< w max 1 , · · · , w max n >, where w min 1 ≤ · · · ≤ w min n and w max 1 ≥ · · · ≥ w max n , we can define OWA fuzzy rough sets as follows: In fact, the definition of a fuzzy relation on an attribute set depends on individual systems. One example of defining relation methods to deal with incomplete information systems is shown in [15]. Further investigation of the combination between fuzzy and rough sets can be found in [5,8,9,21,30].

Rough Selection and Problem Stated
Significant research on instance selection using rough sets model was introduced the work of Caballero et al. [3]. For this purpose, the authors calculated approximations and positive regions for each class in training sets. Instances for training phases are then selected in two ways. In the first method, they try to delete all instances in the boundary. The second employs a nearest neighbourhood algorithm to relabel instances in the boundary region. The issues associated with these approaches were raised in Jensen and Cornelis' study [12]. Another possible limitation is that they just deal with crisp sets of attributes in decision tables.
To deal with the problems identified in the study of Caballero et al., an approach called Fuzzy-Rough Instance Selection (FRIS) has been proposed [12]. The notion of this approach is using memberships of positive regions to determine which instances should be kept and which instances should be discarded for training. First, the method calculates relations among instances and then measures quality of each instance by its positive region membership. An instance will be removed if the membership degree is less than a specified threshold. For the same purpose, another approach of measuring quality of instances was introduced in [27]. Such approaches will undoubtedly refine the positive regions with quality instances.
However, one limitation to be considered is that there is only one threshold used for all classes. In some applications such as medical disease prediction, it is not uncommon to be interested in some specific groups, disease groups for example, rather than the others (healthy groups). When only one threshold is used, once noise is eliminated, valuable instances in interesting classes also disappear.
Some approaches use the advantage of fuzzy-rough to qualify instance sets formed by SMOTE [22,23,28]. Therefore, the issues above may be avoided. The steps of these methodologies can be summarized as follows: 1 Using fuzzy-rough sets to calculate the qualities of every instance and choose instances having high quality. 2 Using SMOTE to create artificial instances from the first step and add to datasets 3 Using fuzzy-rough sets to eliminate low quality instances from datasets after the second step.
There are some differences among these methods. In Ramentol et al.'s study named SMOTE-RSB * [22,23], the first step does not appear. In step 3, the algorithm removes only low quality instances in the artificial instance sets. In Verbiest's research [28], quality measurements are different for step 1 and 3. The first measures performance for imbalanced data while the last deals with balanced information. Thus, they call the algorithm FRIPS-SMOTE-FRBPS (hereinafter referred to as FRPSS in this paper). It is also claimed in [28] that FRPSS produces better training sets compared with SMOTE-RSB * .
It appears that FRPSS is an excellent tool for editing training sets. However, applying three instance selections steps may require considerable computation time compared with other methods.

Multiple Thresholds Fuzzy Rough Instance Selection
As stated in [12], the Fuzzy-Rough Instance Selection (FRIS) algorithm is quite efficient as it eliminates some instances that might affect the positive region membership of the remaining objects. It thus refines the dataset by choosing quality instances for training. However, we also noted some limitations of conventional methods. An example is shown in Figure 1. If we use fuzzy lower approximation as instance qualities and compare these numbers with the threshold = 0.7 (FRIS method), numerous positive instances (minority class) will be removed while almost all negative instances will be remained. The same problem occurs with FRPSS in which an average of approximations is used as a quality measurement. Therefore, in this section, using the advantages of fuzzy rough set instance selection approach, we propose an instance selection method which uses different thresholds for different classes of decision tables. By means of this strategy, SMOTE or other balancing techniques may not be necessary.

MFRIS algorithms
In decision table I = (U, A ∪ {d}), let R a (x, y) and µ Ra (x, y) denote a fuzzy relation and membership function, respectively, between objects x, y ∈ U on attribute a ∈ A. Then the membership function of the relation on an attribute set P ⊆ A is defined as: where T is a t-norm.
In fact, there are many ways to calculate the membership function of a fuzzy relation between two instances. It is also possible to use distance based similarity such as the normalization of Euclidean distance. It depends on the characteristics of each system.

Data:
Xi, Yj, minority and majority classes; tX , tY , the selection thresholds for minority and majority classes. Result: Decision table (S, A ∪ {d}) Calculate membership degree of the relation among objects: µR(x, y); Calculate membership degree of the approximations for each class; Calculate the quality measurement of all instances for their classes; The membership functions of approximation spaces for an object set X ⊆ U on attribute P ⊆ A can be defined by either equations (5) and (6) or (8) and (9). In this study, as mentioned in Section 2, we must note that in the decision table I = (U, A ∪ {d}) is a single label decision table such that f d (x) is a precise value. In fuzzy sets, for a single label decision table, µ Ci (x) > 0 if f d (x) = d i and µ Ci (x) = 0, otherwise.
In selecting learning instances, we first define a function to measure the quality of an instance for each class X in terms of relation R: where α ∈ [0, 1], X is the fuzzy set on X. In the above equation, note that x may or may not belong to X. This is where this approach differs from other approaches. When comparing µ R X (x) for x ∈ X, we can assume that α = 1 as in the study of Jensen et al. [12] and α = 0.5 as in the approach of Verbiest et al. [28]. Now, let t X and t Y be the selection thresholds for minority and majority classes, X i and Y j be the fuzzy sets on minority class X i and majority classes Y j , respectively, R denotes the fuzzy relation on the universal. The set of instances selected for the training phase can be defined as: Then, the algorithm to select instances can be described as shown in Figure 2.
In the first algorithm, depending on labels of instances, the quality measurement of every instance on their classes will be compared with the threshold for minority or majority thresholds. This means that an instance will be deleted from a training set if it is low quality even it is in minority classes. The different thresholds may help us to keep more instance in minority classes if necessary.
In addition, we also propose the second algorithm to select instances. The notion of this method is keeping all instances in minority classes while removing or relabelling some instances in majority classes. To describe the algorithm, we first introduce some definitions.
Let X, Y be the family of minority and majority classes respectively. Let X i , Y j denote fuzzy sets on X i , Y j , respectively. Thresholds for minority and majority classes are t X and t Y , respectively. The set of instances for which labels could be changed can be defined as From the above definition, there are some instances in majority classes whose labels could be changed to a minority class label. The possible minority class set can be determined as follows: Then we can re-calculate the class membership functions of an instance x ∈ S Y →X ∩ Y n as µ Xm (x) = µ Yn (x); µ Yn (x) = 0, where m is defined to satisfy: . Finally, the selected instances for the training phase can be defined as: The algorithm is then described as shown in Figure 3.

Experiment
Experiments were conducted on datasets from KEEL dataset repository [1] and UCI Machine Learning Repository [2] and our lifestyle-diseases dataset. The dataset lifestyle-diseases is raw data of collected health factors such as age, BMI, and waist circumference,... and the outcomes are related to lifestyle diseases. Datasets and their properties are shown in Table 1. In this table, "IR" shows imbalance rates for the number of instances between majority and minority classes. Each dataset is divided into three parts for both minority and majority classes. For each train-test, we used two parts to form training data and the rest were used as testing instances.
Data: X, Y , the family set of minority and majority classes tX , tY , the selection thresholds for minority and majority classes Result: Decision table (S, A ∪ {d}) Calculate membership degree of the relation among objects: µR(x, y); Calculate membership degree of the approximations for each class; Calculate quality measurement of all instances in the majority classes for all classes; SY ← ∅; , where l(a) is the range of value domain on a. In this study, we do not deal with missing values. However, it is possible to define µ Ra (x, y) for incomplete information by following the study in [16].
The fuzzy tolerance relation on a set of attributes can be calculated by a combination of the relation on each attribute by Lukasiewicz's t-norm T = max(x 2 + x 1 − 1, 0). For those which use OWA fuzzy rough sets, the OWA operators are set as follows:  For our approach, we also use OWA fuzzy rough set with above operators and set α = 0.5 to calculate µ R X . That is the same as instances quality measurement in FRIPS-SMOTE-FRBPS in terms of distinguishing instances quality.
Next, selected data are used for training using several classification techniques including k-Nearest Neighbours [7], Support Vector Machine [6], naive Bayes [24] and Decision Trees [26]. It should be noticed that these data classification toolboxes are available within R data mining packages.
To evaluate classification performance in different cases where classifier techniques are used as well as instance selection methods, we use the confusion matrix (Table 2) to calculate the Area Under the ROC Curver(AU C) [11]. AU C provides a single measure for comparing classification on average.
For comparison, we try to obtain the best performance for each instance selection algorithm. FRPSS suggests its own method of threshold optimization. For FRIS and MFRISs, instance measurements are quantized into a set of a limited number of elements. We then use this set as threshold candidates in order to find out if there is a significant increase in performance.
In general, from the average AUC shown in Table 3, we can see that after using the modification of rough set instances selection (both MFRIS1 and MRIS2), the classification performance increases noticeably in almost datasets. It seems that the use of multiple thresholds is more effective with the increase of IR value. Although the original fuzzy rough instance selection approach does not deal with imbalanced data, the increase in performance of MFRISs proves that it is possible to use fuzzy rough sets based instance selection with some criteria to overcome the imbalance issue. Although less consuming time may be an advantage, besides high performance archived, we do not show the time measurements in this version. This is due to the different criteria in choosing thresholds. FRISS, as stated before, has its own method to optimize thresholds. It considers the unique set of all instance measurements as the threshold candidates. This means that the algorithm has to train and test numerous times (nearly as many as the cardinality of training sets). In FRIS and MFRISs, on the other hand, we tried in limited times.
To a comparison of the two algorithms using our approach, MFRIS1 archives better performance for most datasets even though MFRIS2 still improves performance. In the second algorithm, all instances in minority classes are chosen. If such instances do not have sufficient quality, an edited training set from MFRIS2 cannot be better than that from MFRIS1.
When comparing the classifiers, in general, we can see the differences among them in terms of performance evaluation when there is no instance selection method employed (Figure 4, transparency color bars). SVM seems to be sensitive  to imbalance issues, while Bayes provides better performance. This means that for a different type of dataset with respect to imbalance issues, a different choice of classifier may be required. Figure 4 also illustrates the difference in performance increase (dark color bars). SVM does not show a good result for imbalanced data, nevertheless, its performance improves to a greater extent than the rest. In other words, naive Bayes receives less support from instance selection methods. However, the performance of all classifiers generally improves.
Obviously, optimized thresholds are various among classifiers and among datasets. Table 4 shows that SVM needs to remove more instances by using smaller thresholds in comparing with other classifiers. In addition, Table 5 suggests that for some datasets, it is necessary to edit majority classes by discarding many instances (high majority thresholds) while moving some instances to minority classes.

Conclusion and future work
In this paper, we presented a modification of fuzzy-rough instance selection using multiple thresholds. Retaining the advantages of fuzzy-rough sets when selecting instances, it can measure the quality of instances for training phase in order to select valuable instances. It also deals with imbalanced data by using different thresholds between minority and majority classes. Some experiments show that these approaches can improve performance considerably. They can also be competitive with state-of-the-art methods. Furthermore, this notion can potentially applied to modify current rough set based learning methods in order to eliminate the instance selection step.
This paper mainly proves the robustness of using multiple thresholds in fuzzyrough sets based instance selection approach. No algorithm for selecting valuable thresholds is suggested. Therefore, in further research we intend to study a method of optimizing thresholds and will endeavour to discover a better instance quality measurement function.