{'fraction_non_alphanumeric': 0.029602258573445155, 'fraction_numerical': 0.009964294610977332, 'mean_word_length': 4.864864864864865, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hierarchical Web Page Classification Based on a Topic Model and Neighboring Pages Integration

Most Web page classification models typically apply the bag of words (BOW) model to represent the feature space. The original BOW representation, however, is unable to recognize semantic relationships between terms. One possible solution is to apply the topic model approach based on the Latent Dirichlet Allocation algorithm to cluster the term features into a set of latent topics. Terms assigned into the same topic are semantically related. In this paper, we propose a novel hierarchical classification method based on a topic model and by integrating additional term features from neighboring pages. Our hierarchical classification method consists of two phases: (1) feature representation by using a topic model and integrating neighboring pages, and (2) hierarchical Support Vector Machines (SVM) classification model constructed from a confusion matrix. From the experimental results, the approach of using the proposed hierarchical SVM model by integrating current page with neighboring pages via the topic model yielded the best performance with the accuracy equal to 90.33% and the F1 measure of 90.14%; an improvement of 5.12% and 5.13% over the original SVM model, respectively.


INTRODUCTION
Due to the rapid growth of Web documents (e.g., Web pages, blogs, emails) on the World Wide Web (WWW), Web page classification has become one of the key techniques for managing and organizing those documents, e.g., as document filtering in information retrieval. Generally, Web page classification applies the technique of text categorization, which uses the supervised machine learning algorithms for learning the classification model [1,2]. Most previous works on Web page classification typically applied the bag of words (BOW) model to represent the feature space. Under the BOW model, a Web page is represented by a vector in which each dimension contains a weight value (e.g., frequency) of a word (or term) occurring in the page. The original BOW representation, however, is unable to recognize synonyms from a given word set. As a result, the performance of a classification model using the BOW model could become deteriorated.
In this paper, we apply a topic model to represent the feature space for learning the Web page classification model. Under the topic model concept, words (or terms), which are statistically dependent, are clustered into the same topics. Given a set of documents D consisting of a set of terms (or words) W, a topic model generates a set of latent topics T based on a statistical inference on the term set W. In this paper, we apply the Latent Dirichlet Allocation (LDA) [3] algorithm to generate a probabilistic topic model from a Web page collection. A topic model can help capture the hypernyms, hyponyms and synonyms of a given word. For example, the words "vehicle" (hypernym) and "automobile" (hyponym) would be clustered into the same topic. In addition, the words "film" (synonym) and "movie" (synonym) would also be clustered into the same topic. The topic model helps improve the performance of a classification model by (1) reducing the number of feature dimensions and (2) mapping the semantically related terms into the same feature dimension.
In addition to the concept of topic model, our proposed method also integrates some additional term features from neighboring pages (i.e., parent, child and sibling pages). Using some additional terms from neighboring pages could help increase more evidence for learning the classification model [4,5]. We used the Support Vector Machines (SVM) [6,7] as the classification algorithm. SVM has been successfully applied to text categorization tasks [6,7,8,9]. SVM is based on the structural risk minimization principle from computational theory. The algorithm addresses the general problem of learning to discriminate between positive and negative members of a given class of n-dimensional vectors. Indeed, the SVM classifier is designed to solve only the binary classification problem [7]. In order to manage the multi-class classification problem, many researches have proposed hierarchical classification methods for solving the multi-class problem. For example, Dumais and Chen proposed the hierarchical method by using SVM classifier for classifying a large, heterogeneous collection of web content. The study showed that the hierarchical method has a better performance than the flat method [10]. Cai and Hofmann proposed a hierarchical classification method that generalizes SVM based on discriminant functions that are structured in a way that mirrors the class hierarchy. The study showed that the hierarchical SVM method has a better performance than the flat SVM method [11].
Most of the related work presented a hierarchical classification method by using different approaches. However in previous works, the bag of words (BOW) model is used to represent the feature space. In this paper, we propose a new hierarchical classification method by using a topic model and integrating neighboring pages. Our hierarchical classification method consists of two phases: (1) feature representation and (2) learning classification model. We evaluated among three different feature representations: (1) applying the simple BOW model on current page, (2) applying the topic model on current page, and (3) integrating the neighboring pages via the topic model. To construct a hierarchical classification model, we use the class relationships obtained from a confusion matrix of the flat SVM classification model. The experimental results showed that by integrating the additional neighboring information via a topic model, the classification performance under the F1 measure was significantly improved over the simple BOW model. In addition, our proposed hierarchical classification method yielded a better performance compared to the SVM classification method.
The rest of this paper is organized as follows. In the next section we provide a brief review of Latent Dirichlet Allocation (LDA). Section 3 presents the proposed framework of hierarchical classification via the topic model and neighboring pages integration. Section 4 presents the experiments with the discussion on the results. In Section 5, we conclude the paper.

II. A REVIEW OF LATENT DIRICHLET ALLOCATION
Latent Dirichlet Allocation (LDA) has been introduced as a generative probabilistic model for a set of documents [3,12]. The basic idea behind this approach is that documents are represented as random mixtures over latent topics. Each topic is represented by a probability distribution over the terms. Each article is represented by a probability distribution over the topics. LDA has also been applied for identification of topics in a number of different areas such as classification, collaborative filtering [3] and content-based filtering [13].
Generally, an LDA model can be represented as a probabilistic graphical model as shown in Figure 1 [3]. There are three levels to the LDA representation. The variables α and β are the corpus-level parameters, which are assumed to be sampled during the process of generating a corpus. α is the parameter of the uniform Dirichlet prior on the per-document topic distributions. β is the parameter of the uniform Dirichlet prior on the per-topic word distribution. θ is a document-level variable, sampled once per document. Finally, the variables z and w are word-level variables and are sampled once for each word in each document. The variable N is the number of word tokens in a document and variable M is the number of documents. The LDA model [3] introduces a set of K latent variables, called topics. Each word in the document is assumed to be generated by one of the topics. The generative process for each document w can be described as follows:   Figure 2 illustrates the proposed hierarchical classification framework which consists of two phases: (1) feature representation for learning the Web page classification models, (2) learning classification models based on the Support Vector Machines (SVM). In our proposed framework, we evaluated among three different feature representations: (1) applying the simple BOW model on current page, (2) applying the topic model on current page, and (3) integrating the neighboring pages via the topic model. After the feature representation process, we use the class relationships obtained from a confusion matrix of the flat SVM classification model for building a new hierarchical classification method.

A. Feature Representation
The process for feature representation can be explained in details as follows.
• Approach 1 (BOW): Given a Web page collection consists of an article collection which is a set of m documents denoted by D = {D 0 , …, D m−1 }. In the process of text processing is applied to extract terms. Given a set of terms is represented W = {W 0 , …, W k-1 }, where k is the total number of terms. Each term is provided with certain weight w i , which the weight of each term is assigned with term frequency. The set of terms is then filtered by using the feature selection technique, information gain (IG) [1]. Once the term features are obtained, we apply the Support Vector Machines (SVM) to learn the classification model. The model is then used to evaluate the performance of category prediction.
• Approach 2 (TOPIC_CURRENT): Given a Web page collection consisting of an article collection which is a set of m documents denoted by D = {D 0 , … , D m−1 }. The process of text processing is applied to extract terms. The set of terms is then generated by using the topic model based on the LDA algorithm. The LDA algorithm generates a set of n topics denoted by T = {T 0 , … , T n−1 }. Each topic is a probability distribution over p words denoted by w is a probabilistic value of word j assigned to topic i.
Based on this topic model, each document can be represented as a probability distribution over the topic set T, i.e., j is a probabilistic value of topic j assigned to document i. The output from this step is the topic probability representation for each article. The Support Vector Machines (SVM) is also used to learn the classification model.

• Approach 3 (TOPIC_ INTEGRATED):
The main difference of this approach from Approach 2 is we integrate the additional term features obtained from the neighboring pages to improve the performance of Web page classification.
The process of integrating the neighboring pages is explained as follows. Figure 3 shows three types of neighboring pages, parent child and sibling pages. Given a Web page (i.e., current page), there are typically incoming links from parent pages, outgoing links to child pages and links from its parent pages to sibling pages. A parent child and sibling pages are collectively referred to as the neighboring pages. Using the additional terms from the neighboring pages could help increase more evidence for learning the classification model.
In this paper, we vary a weight value of neighboring pages from zero to one. A weight value equals to zero means the neighboring pages are not included for the feature representation. Under this approach, terms from different page types (i.e., current, parent, child and sibling) are first transformed into a set of n topics (denoted by T = {T 0 ,..., T n-1 }) by using the LDA algorithm. The weight values from 0 to 1 are then multiplied to the topic dimension T i of parent, child and sibling pages. The combined topic feature vector by integrating the neighboring topic vectors with adjusted weight values can be computed by using the algorithm listed in Table  1.    The INP algorithm that we present in this paper incorporates term features obtained from the neighboring pages (i.e. parent, child and sibling pages) into the classification model. Using additional terms from the neighboring pages could help increase more evidence for learning the classification model. In this algorithm, we propose a function for varying the weight values of terms from parent pages (PDT), child pages (CDT) and sibling pages (SDT). The probability values from all neighboring pages are integrated with the current page (CurDT) to form a new integrated matrix (IDT).
The process of algorithm begins with the results from the LDA model; that is document-topic matrices from all page types. The algorithm starts by gathering data from documenttopic matrices (CurDT, PDT, CDT, SDT) using getPValue function. All P-values of the document-topic matrices are then multiplied by the weight values of each document-topic matrix except for the current page matrix. Finally all P-values from four matrices are summed up and then sent to IDT using setPValue function. After the integrating process, we use the IDT matrix for learning the classification model.

B. Classification Model
Three different feature representation approaches are used as input to classifiers. In this paper, we propose two methods for building the classification models: (1) Model 1: we adopt the SVM to classify feature and (2) Model 2: we presented a new hierarchical classification method by using the class relationships obtained from a confusion matrix for learning a classification model. Each method is described in details as follows.
• Model 1 (SVM): We used the SVM for learning a classification model. The SVM is the machine learning algorithm proposed by Vapnik [7]. The algorithm constructs a maximum margin hyperplane which separates a set of positive examples from a set of negative examples. In the case of examples not linearly separable, SVM uses a kernel functions to map the examples from input space into high dimensional feature space. Using a kernel function can solve the non-linear problem. In our experiments, we used a polynomial kernel. We implemented the SVM classifier by using the WEKA 1 library.

• Model 2 (HSVM):
The proposed method is based on SVM classifier, which uses the class relationship obtained from a confusion matrix for building a hierarchical SVM (HSVM). A confusion matrix shows the number of correct and incorrect predictions made by the model compared with the actual classifications of the test data. The size of confusion matrix is m-by-m, where m is the number of classes. Figure 4 shows an example of a confusion matrix from Approach 3 built on a collection of articles obtained from the Wikipedia Selection for Schools. In a confusion matrix, the row corresponds to the actual classes, and the column corresponds to the prediction classes. In this example, for class art, the model makes the correct prediction equal to 49 instances and incorrect prediction into class citizenship (c) for 1 instance and into class design and technology (e) for 5 instances. We used the confusion matrix for constructing a hierarchical structure. First, we need to transform the confusion matrix into a new symmetric matrix, called average pairwise confusion matrix (APCM) by computing average values of pairwise relationships between classes in a confusion matrix (CM). The process of transforming CM into APCM can be explained as follows. Given a confusion matrix CM = [v a,p ], where a denotes each row corresponding to actual classes and p denotes each column corresponding to the prediction classes. For the correct prediction, i.e., a equals to p in CM, we set the value equal to 0 in APCM. If a is not equal to p, i.e., incorrect prediction, we compute an average value of v a,p and v p,a for a pairwise confusion value at this position. We applied this calculation method for every row and column. For example, in Figure 4, v 0,0 = 49, a is equal to p (a correct prediction), v 0,0 is set equal to 0 in APCM. For v 0,2 = 1, where a = 0, p = 2 (a is not equal to p), an average pairwise confusion value of v 0,2 and v 2,0 is equal to 1. The final result of an average pairwise confusion matrix computation is shown in Figure 5. The computation of an average pairwise value is summarized by the following equation: Once the average pairwise confusion matrix (APCM) is obtained, we construct a dendrogram based on the single link algorithm of hierarchical agglomerative clustering (HAC) [14,15]. Single link clustering is known to be confused by nearby overlapping clusters which merge two clusters with the smallest minimum pairwise distance [14]. To construct our hierarchical classification structure, we adopt the single link algorithm to merge two clusters by selecting maximum average pairwise value in a confusion matrix. We first select a pair of classes which has the maximum average pairwise value in APCM to a dendrogram and select the next highest average pairwise value and go on with this process until all classes are selected into the dendrogram. The final result of dendrogram is shown in Figure 6. For example, an average pairwise value between class f and o is 21.5, the highest value in APCM, therefore class f and class o are selected as the first pair in the dendrogram. The second highest value is 19, this value is an average pairwise value between class h and m, therefore class h and class m are selected as the second pair. The third highest value is 15 between class g and o. However, class o is already paired with class f. Therefore, we take only class g to combine with class f and class g nodes. We perform this process for all remaining classes. Finally, we obtain a complete dendrogram for constructing the hierarchical classification model. The hierarchical classification models are constructed from bottom-up level. With this hierarchical classification structure, classes with lower confusion values are classified before classes with higher confusion. The hierarchical classification model could help improve the performance of multi-class classification method.

A. Web page collection
In our experiments, we used a collection of articles obtained from the Wikipedia Selection for Schools which is available from the SOS Children's Villages Web site 2 . There are 15 categories: art, business studies, citizenship, countries, design and technology, everyday life, geography, history, IT, language and literature, mathematics, music, people, religion and science. The total number of articles is 4,625. Table 2 lists the first-level subject categories available from the collection. Organizing articles into the subject category set provides users a convenient way to access the articles on the same subject. Each article contains many hypertext links to other articles which are related to the current article.

B. Experiments
We used the LDA algorithm provided by the linguistic analysis tool called LingPipe 3 to run our experiments. LingPipe is a suite of Java tools designed to perform linguistic analysis on natural language data. In this experiment, we applyed the LDA algorithm provided under the LingPipe API and set the number of topics equal to 200 and the number of epochs to 2,000.For text classification process, we used WEKA, an open-source machine learning tool, to perform the experiments.

C. Evaluation Metrics
The standard performance metrics for evaluating the text classification used in the experiments are accuracy, precision, recall and F1 measure [16]. We tested all algorithms by using the 10-fold cross validation. Accuracy, precision, recall and F1 measure are defined as: 2  where Accuracy represents the percentage of correct predictions in total predictions. Precision (P) is the percentage of the predicted documents for a given category that are classified correctly. Recall (R) is the percentage of the documents for a given category that are classified correctly. F1 measure is a single measure that tries to combine precision and recall. F1 measure ranges from 0 to 1 and the higher the better.

D. Experimental results
We started by evaluating the weight values of neighboring pages under Approach 3. Table 3 shows the results of combination the weight value of neighboring pages on our algorithm. For the SVM model, the best combination of neighboring pages with the accuracy equal to 85.21% and the F1 measure of 0.8501 by weight of parent pages, child pages and sibling pages equal to 0.4, 0.0 and 0.3, respectively and for the HSVM model has the best combination of neighboring pages with the accuracy equal to 90.33% and the F1 measure of 0.9014 by weight the same SVM model. The results showed that using information from parent pages and sibling pages are more effective than child pages for improving the performance of a classification model.   Table 5 shows the experimental results of three feature representation approaches by using two models between the SVM model and the hierarchical SVM (HSVM) model for the learning classification model. From this table, the approach of integrating current page with neighboring pages via the topic model (TOPIC_INTEGRATED) yielded a higher performance compared to applying the topic model on current page (TOPIC_CURRENT) and application of the BOW model. The HSVM classification model yielded a higher performance compared to the SVM classification model in all three feature representation approaches.
The results of classification model based on the SVM model, applying the TOPIC_CURRENT approach helped improve the performance over the BOW by 17.2% based on the F1 measure and applying the TOPIC_INTEGRATED approach, yielded the best performance with the F1 measure of 85.01%; improvement of 23.81% over the BOW model. For the learning classification model based on the HSVM model, applying the TOPIC_CURRENT approach helped improve the performance over the BOW by 3.88% based on the F1 measure and applying the TOPIC_INTEGRATED, yielded the best performance with the F1 measure of 90.14%; improvement of 5.11% over the BOW model. The approach of integrating current page with the neighboring pages via the topic model (TOPIC_INTEGRATED) and using the HSVM model, however, yielded the best performance with the F1 measure of 90.14%; improvement of 5.13% over the TOPIC_INTEGRATED approach by using the SVM model. Thus, integrating the additional neighboring information, especially from the parent pages and sibling pages, via a topic model could significantly improve the performance of a classification model. The reason is due to the parent pages often provide terms, such as in the anchor texts, which provide additional descriptive information of the current page.

V. CONCLUSIONS
To improve the performance of Web page classification, we proposed a new hierarchical classification method based on a topic model and by integrating the additional term features obtained from the neighboring pages to improve the performance of Web page classification. We applied the topic model approach based on the Latent Dirichlet Allocation algorithm to cluster the term features into a set of latent topics. Terms assigned into the same topic are semantically related. Our hierarchical classification method consists of two phases: (1) feature representation by using a topic model and integrating neighboring pages, and (2) hierarchical Support Vector Machines (SVM) classification model constructed from a confusion matrix. From the experimental results, the approach of integrating current page with the neighboring pages via the topic model yielded a higher performance compared to applying the topic model on current page and applying the BOW model. For learning classification model, the hierarchical SVM classification model yielded a higher performance compared to the SVM classification model in all three feature representation approaches and integrating current page with the neighboring pages via the topic model approach, however, yielded the best performance with the F1 measure of 90.14%; improvement of 5.11% over the BOW model. The approach of integrating current page with the neighboring pages via the topic model and using the hierarchical SVM classification model yielded the best performance with the accuracy equal to 90.33% and the F1 measure of 90.14%; an improvement of 5.12% and 5.13% over the original SVM model, respectively.