{'fraction_non_alphanumeric': 0.017580144777662874, 'fraction_numerical': 0.0, 'mean_word_length': 5.088050314465409, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Machine Learning Approach to Webpage Content Exraction

Apart from the main content, a webpage usually also contains boilerplate elements such as navigation panels, advertisements and comments. These additional elements are typically not related to the actual content and can be treated as noise that needs to be removed properly to improve the user’s reading experience. This is a difficult problem as HTML is loose in semantics and flexible in structure. In this paper, we model the webpage content extraction problem as a classification problem and employ machine learning method to solve it. For each text block in the HTML document, we select a set of relevant features, based on which an SVM classifier is used to predict whether this text block is content or non-content. The results show that our approach can achieve performance comparable to some popular existing algorithms in this field. Keywords—text mining, content extraction, web semantic, algorithm