{'fraction_non_alphanumeric': 0.026415491475478847, 'fraction_numerical': 0.005788255104188592, 'mean_word_length': 4.939375, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Extracting Information-rich Part of Texts using Text Denoising

The aim of this paper is to report on a novel text reduction technique, called Text Denoising, that highlights information-rich content when processing a large volume of text data, especially from the biomedical domain. The core feature of the technique, the text readability index, embodies the hypothesis that complex text is more information-rich than the rest. When applied on tasks like biomedical relation bearing text extraction, keyphrase indexing and extracting sentences describing protein interactions, it is evident that the reduced set of text produced by text denoising is more information-rich than the rest.


Introduction
Often, to test a method's scalability as well as its performance across genres of texts, there is a need to process large volumes of text data in many disciplines of NLP, be it textual relation extraction, summarization or meta-tagging. It has been reported by many researchers [?][?] that machine learning as well as rule-based approaches show improvements over their benchmarks with increased training data. However, the use of large volume of data can create several bottlenecks. One is technical-processing large data, like that from biomedical texts, slows down many algorithms; another is even more important-algorithms can exhibit a decreased accuracy because of the noise, which are irrelevant or redundant data for a given classification task, added by information-poor parts of texts.
There are several statistics, like word-level feature tf-idf and sentence-level feature sentence position, that help identify information richness. Although the degree of use shows their popularity, these features have some serious limitations. For example, tf-idf computes document similarity directly in the word-count space which may be slow for large vocabularies and sentence position is useful for summarization but is superficial in relation extraction. In other words, they are either task-specific and/or domain-specific measures.
Text readability has multivariate features that consider many attributes like length of paragraph, words and sentences, and number of polysyllabic and monosyllabic words. In this paper, I report a text reduction technique called Text Denoising that reduces text data based on text readability, especially from the biomedical domain, to that which is more information-rich by removing most of the noise. The reduced text is also expected to be task-independent and informative enough to improve accuracy of NLP tools across disciplines.

Proposed Method
Among text readability scores, the following five measures are considered as yardsticks-Fog Index ( [?]. The choice of using text readability as an information richness statistic is motivated by the results of an experiment by Duff and Kabance [?]. In their experiment, a passage with no more than two phrases were converted into primer prose and FI was applied to test its readability. They found that the score was low (i.e., the prose was extremely easy to read). The authors concluded that easy texts obscure the relationships and ideas as they de-emphasize both. In contrast, difficult texts emphasize relationships and ideas yielding low readability. I suggest that the describing of biomedical relations, meta information, etc. lengthens sentences as well as increases the use of polysyllabic words which are the two principal components of many of the readability indexes.
Both rule-based and machine learning-based versions of Text Denoising are based on this principle that use text readability as a key feature and applies it at the sentence-level to identify those sentences within a text, called denoised text, where content information, such as biomedical relations, is more likely to occur. The rest of the text is called noise text. I am interested to observe the effect of using text denoising on different tasks and genres of text.

Text Denoising on Relation Extraction
I developed a corpus of 24 texts that describe four pairs of related MeSH C and MeSH D concepts reported by Perez et al. [?]. I applied the rule-based version of text denoising on these texts to extract related biomedical concepts. The only rule I set for this task was to extract 30% of the low-readability sentences from the texts according to their FI score. This threshold is termed as the denoising threshold and the texts extracted are called denoised texts; the rest is called noise text. This threshold point was set heurisitically considering the stability in the frequency of appearance of the related concepts in the corpus. Other than 30%, the results with different denoising thresholds ranging from 10% to 50%, however, was not satisfactory. I ranked the pairs of concepts present in the denoised texts using their frequency. Most of the concept pairs with higher ranks, however, did not contain any semantic relations according to UMLS semantic relation network. Therefore, I re-ranked the pairs according to their positive predictive value (PPV) (similar to precision measure used in information retrieval evaluation tasks) and sensitivity. The pairs of concepts found from this re-ranking showed a convincing accuracy of 75% (ratio of semantically related Glutamate-CA4 Yes 7 Increase-Glutamate Yes 8 10min-Glutamate No 9 Ischemia-5min Yes 9 Glutamate-5min No Later, I performed an experiment with four other readability scores mentioned in Section 2 on the same corpus. A comparative result showed that FI outperformed the other indexes by extracting more meaningful relations [?]. I also analyzed the performance of the indexes considering the performance of FI as a benchmark. Table 2a and 2b show that the SMOG index is a close second to FI followed by FKRI, while FRES and FORCAST performed poorly. It can also be noted that the SMOG index, like FI, uses the core measure of complex words which reveals the fact that the measure of complex word fits best for text denoising and biomedical relation extraction.   the indexers with their classifiers induced from denoised training data on three datasets, namely FAO-780, CERN-290 and NLM-500. These datasets are composed of texts from the domains of agriculture, physics and biomedical science. I compared the result with their benchmark performances that were achieved by using the full-text training data. Convincingly, in a 10-fold cross validation experiment, both KEA and KEA++, with their classifiers induced from denoised training data, outperformed their respective benchmark F-scores [?]. Maui, on the other hand, had mixed results and its denoised text induced classifier performs comparably with its benchmark [?]. The F-Scores are listed in Table 3a, 3b and 3c where a t-value greater than or equal to 2.26 indicates the statistical significance of the results at 95% confidence. Of note, unlike the fixed denoising threshold of 30% for relation extraction, I found that to get bias-free classifiers for the indexers, the denoising threshold point needed to be varied (usually between 30%-70%) for different genres of texts. This outcome confirms that the rule to decide the amount of text to be extracted from texts substantially depends for different writing styles.

Text Denoising on Extracting Protein Relations bearing Sentences
In an attempt to eliminate the denoising threshold which depends on writing style (Section 4), I decided to develop a machine learning version of text denoising. The classification task in hand was to annotate sentences of a set of texts   ?] are used to assign binary labels to each sentence of these corpora depending on the presence of any protein relations. Having realized after this assignment that the classes are negatively skewed (almost doubled the positive labels), synthetic positive samples are produced using SMOTE [?] where the minority class is over-sampled by taking each minority class sample and introducing synthetic examples along the line segments joining any/all of the k, which is five in our setup, minority class nearest neighbors. From initial results, I found that many features were highly correlated with each other but had low correlation with the class. Therefore, I used a wrapper method to select a set of bias-free features. However, I observed that this set of features varies for different corpora. Table 4 shows the precision, recall and F-Score of text denoising in a 10-fold cross validation setup, considering the highly agreed upon annotation of RelEx and WRelEx as the gold standard. It can be noted that the outcome of this experiment without using SMOTE was not satisfactory as the F-scores were under 80%.

Conclusion and Future Work
The proposed text denoising method performed much the same on several tasks and kinds of texts: the reduction of texts according to the readability improved relation mining, keyphrase indexing and extracting sentences that describe protein relations. This result strongly suggests that sentences that are difficult to read are more information-rich than the rest. The effect of text denoising is yet to be examined for text categorization and summarization. I am currently investigating the effect of readability on e-mail spam detection. The results so far are interesting as I am labelling spam and ham based on the readability of e-mail text content only (i.e., without looking at the mail header). Also, I intend to train benchmark summarizers with denoised texts and see how they perform against gold standard summaries.