{'fraction_non_alphanumeric': 0.03700383485233329, 'fraction_numerical': 0.01673390109069177, 'mean_word_length': 5.1107729762629335, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sentence Classification with Imbalanced Data for Health Applications

Identifying and extracting reports of medications, their abuse or adverse effects from social media is a challenging task. In social media, relevant reports are very infrequent, causes imbalanced class distribution for machine learning algorithms. Learning algorithms typically designed to optimize the overall accuracy without considering the relative distribution of each class. Thus, imbalanced class distribution is problematic as learning algorithms have low predictive accuracy for the infrequent class. Moreover, social media represents natural linguistic variation in creative language expressions. In this paper, we have used a combination of data balancing and neural language representation techniques to address the challenges. Specifically, we participated the shared tasks 1, 2 (all languages), 4, and 3 (only the span detection, no normalization was attempted) in Social Media Mining for Health applications (SMM4H) 2020 (Klein et al., 2020). The results show that with the proposed methodology recall scores are better than the precision scores for the shared tasks. The recall score is also better compared to the mean score of the total submissions. However, the F1-score is worse than the mean score except for task 2 (French).


Introduction
Advances in representation learning that attempts to automatically learn features for natural language processing (Young et al., 2018) present the possibility of utilizing social media (i.e. Twitter) data source for public health applications such as health monitoring and surveillance. Several ethical, legal and methodological challenges need to be addressed that are unique to Twitter data source (Ahmed et al., 2017). The ongoing shared tasks in Social Media Mining for Health applications (SMM4H) define evolving challenges specific to the Twitter data source for health domain (Weissenbacher et al., 2019). To address the methodological challenges, in recent years, several techniques have been proposed based on the SMM4H shared tasks (Sarker et al., 2018;Weissenbacher et al., 2019).
Epidemiologists intend to detect mentions of health issues related to medications early on Twitter. The adverse effect of medications is one of the leading causes of post-therapeutic deaths (Saha et al., 2018). One of the challenges of detecting real reports on medications, their abuse or adverse effects is to distinguish the relevant true reports from other general statements, news, and institutional advice. Reports on health issues (i.e. abuse or adverse effects of a medication) in social media are rare instances (i.e. very small percentage contain relevant information) (Weiss, 2004;Batista et al., 2004). This causes a major machine learning methodological challenges called imbalanced learning problem (He and Garcia, 2009;Ling and Sheng, 2010). The learning problem happens in the presence of underrepresented data and severely skewed class distribution, the situation where one of the class categories comprises a significantly larger proportion of the dataset than the other classes. Imbalanced class distribution is a practical issue in most real-world datasets (e.g. fraud detection, disease detection) and complicates learning when the identification of the minority class is of specific importance. This is a general problem for health domain with medical data utilizing machine learning methods (Rahman and Davis, 2013).
When dealing with imbalanced data, model evaluation is different than typical class-balanced loss based machine learning framework (Cui et al., 2019;Marchand and Strawderman, 2020). Models that trained by minimizing errors on imbalanced datasets, tend to frequently predict the majority class; achieving high overall accuracy in such cases can be misleading. As class imbalance is a widespread issue, multiple techniques have been developed that help alleviate the issue (Buda et al., 2018;Haixiang et al., 2017), by either adjusting the model (e.g. changing the performance metric) or changing the data distribution (e.g. oversampling the minority class or undersampling the majority class). In this paper, we have explored novel strategies to handle extreme imbalanced class distribution.
Another challenge is representing informal Twitter text efficiently for health monitoring. Tweets contain misspelled unnormalized health concepts, expressed in a noisy, ungrammatical, multilingual, and ambiguous way. Moreover, creative and colloquial language expressions are prevalent in Twitter text. In this paper, advanced preprocessing and feature learning techniques were utilized to efficiently capture syntactic and semantic regularities of the substantially informal Twitter data. Overall, several techniques were explored to address challenges in Twitter data source for health monitoring and surveillance. The paper is organized as follows: section 2 describes data and problem statement, section 3 and 4 include the methodology used to model the datasets, and section 5 contains a discussion of the obtained results to understand the challenges specifically related to imbalanced dataset.

Problem Statement and Data Description
In this paper, six datasets from the Social Media Mining for Health application (SMM4H) 2020 Shared tasks were used for health monitoring and surveillance challenges. All the shared task datasets are labeled and can be modeled in one of three popular types of supervised classification problems. Binary (B) classification is the task of classifying the elements of a given set into two groups whereas multiclass (M) classification generalizes into more than two groups. Span detection (D) and normalization (N) task is defined as D+N which can be modeled as multi-class classification task followed by named-entity recognition modeling (Nadeau and Sekine, 2007;Lample et al., 2016).  Table 1: Brief data description: rows correspond to the datasets and columns correspond to various attributes related to the datasets. The first column describes the task with an identifier. The second column denotes a classification type for the task (i.e. binary, multi-class and span-detection). The third column describes the imbalance ratio (IR), defined as the ratio of the minority class examples to the total number of examples, in percentage. The fourth column (# (*)) has the class/label distribution over the training dataset where #(*) denotes the number of Tweets labeled with *. Each dataset is comprised of three splits: training, validation and testing set. The fifth (i.e. #Train), sixth (i.e. #Valid.) and seventh (i.e. #Test) column correspond to the number of Tweets available for training, validation and testing set. Table 1 shows a brief description of the datasets from the SMM4H 2020 shared tasks (Klein et al., 2020). Task 1 requires distinguishing between two classes of Tweets and modeled as a binary classification task. Positive Tweets that mention a medication or dietary supplement are labeled as "1" and negative Tweets that do not mention are labeled as "0". The dataset consists of Tweets posted by 112 women during pregnancy, with approximately 0.26% (see column IR%) of the training Tweets mentioning a medication or dietary supplement. The data set represents an extremely 1 imbalanced class distribution.
Task 2 is another binary classification task that involves classifying Tweets based on the mentions of adverse effect (AE) of a medication. This task includes distinct sets of Tweets posted in three languages: English (En), French (Fr) and Russian (Ru). In Tab. 1, rows corresponding to task id T2 (En), T2 (Fr) and T2 (Ru) refer to sub-tasks in three languages. Tweets that mention an adverse effect of medications are labeled as "1" and those that do not have mention are labeled as "0". The tasks require taking into account subtle linguistic and semantic variations between AEs and indications (i.e. the reason for using the medication). The T2 (Fr), T2 (En) and T2 (Ru) datasets represent moderately imbalanced class distribution with IR of 1.61%, 9.25% and 8.75% respectively.
Task 3 involves detecting the span of Tweet containing an adverse effect (AE) of a medication and then mapping the extracted AE to a standard concept identifier (ID) in the MedDRA vocabulary (preferred terms). The training data includes Tweets that report or indicate an AE that are labeled as "1" and those that do not mention are labeled as "0". The detection task thus requires a model to distinguish between AEs and indications. The class distribution of the dataset is balanced (IR=51.20%) for the detection task. The normalization task involves classification to multiple classes where each class can be defined as a standard MedDRA ID. We have not attempted the normalization part of the task and left for future work.
Task 4 requires distinguishing between more than two classes of Tweets and can be modeled as a multi-class classification problem. The task involves distinguishing among Tweets that mention at least one prescription opioid, benzodiazepine, atypical anti-psychotic, central nervous system stimulant or GABA analogue. Tweets that report potential abuse/misuse are labeled as "A" from those that report non-abuse/-misuse consumption which are labeled as "C", merely mention of the medication are labeled as "M", and unrelated are labeled as "U". The task has moderately imbalanced class distribution with IR of 15.99%.

Methodology for Classification Tasks
We have developed systems for shared tasks 1, 2 (all languages), 4, and 3 (only the span detection, no normalization was attempted). The main focus was towards the imbalanced classification tasks. In this section, we will describe the methods used for five classification tasks and in the next section, we will describe the method developed for the span-detection part of task 3. The models were trained on the training dataset and evaluated on validation dataset; test data was not available when the methods were being developed. Each solution of the binary and multi-class classification task can be subdivided into four common steps and these steps will be elaborated in the following subsections.

Preprocessing
Tweet language processing is challenging compared to the standard text found in the news, journals and books (Balahur, 2013). For example, Twitter users use an informal language (Tan et al., 2015) that uses special expressions, such as "lol", "omg", emoticons (Derks et al., 2008), and emphasize or exaggerate the underlying meaning of the root word by using stretched words like 'heellllp' or 'heyyyyy'. Using the stretched word is common in spoken language but Tweets can have them in written format. The traditional mainstream natural language processing tools were not designed to include the syntactic regularities of informal language (Kong et al., 2014). Additionally, Twitter language has specific metadata, such as "RT" defines Tweets that reposted by other users, the markup of topics using the hashtag ("#") sign and defining other Twitter users by "@" sign. In this paper, we have considered the Tweet specific characteristics when preprocessing the Tweets so that the language can be normalized -converting them into a more standard form of language -efficiently. Normalization of Twitter posts enable us to apply standard natural language processing (NLP) techniques more effectively.
A basic normalization (Kaufmann and Kalita, 2010;Declerck and Lendvai, 2015;Beckley, 2015) was employed as part of advanced preprocessing step for the datasets. The preprocessing stage includes the following steps: separating hyperlinks from the adjacent text, normalize twitter-specific tokens, extracting text from '*' (e.g. *good* > good), replacing & symbol, lower-casing the text, normalize multiple occurrences of vowels and consonants, normalize emojis and numbers, spiting 'number' and 'emoji' when adjacent to text, removing non-alphanumeric characters, removing very long words >= 15 and short words < 2 to reduce sparsity, removing multiple sequential occurrences of the same token.

Under-sampling
As one of the main focus of this paper is to model imbalanced dataset, we have utilized random resampling (Napierała et al., 2010;Johnson and Khoshgoftaar, 2019) to balance the class distribution. After Tweet preprocessing, we have used random under-sampling of the majority class. Oversampling is another approach to balance the training dataset, however, often oversampling minority class instances does not fully balance the training data whereas under-sampling of majority class is found to be better at balancing the training data (Jamil, 2017). We have left experiments involving oversampling for future works.
For extremely imbalanced class distribution in task 1, the under-sampling was done in two steps. First, we have utilized an informed under-sampling technique based on a pre-trained named-entity recognizer (NER) (Andriy Mulyar and McInnes, 2018) trained on clinical notes 2 . The NER model was used to extract and remove the Tweets that have medical named entities represented by informal expressions (e.g. 'lol').
After that, random under-sampling of the majority class was done on the remaining Tweets. The balanced dataset size is twice the number of the positive label in the dataset. For example, task 1 has a total of 146 positive examples (i.e. minority class) and after applying the random under-sampling, the negative examples (i.e. majority class) reduced down to 146. The resulting balanced training dataset size is 292 (i.e. 146 + 146). This part of random down-sampling was applied for all the datasets with IR < 50%. A python toolbox, imbalanced-learn (Lemaître et al., 2017), was used to re-balance the class distribution.

Sentence Embedding
Sentence embedding methods attempt to encode a variable-length input sentence into a fixed-length vector. While preprocessing and normalization helps in better syntactic representation (Kaufmann and Kalita, 2010;Kong et al., 2014), sentence embeddings were used to improve the semantic representation. Moreover, sentence embeddings have been used in sentence classification to address the class imbalance problem (Madabushi et al., 2019). In recent years, several sentence embedding methods, exploration of semantic properties of resulted embeddings and impact of embedding on the downstream application have been proposed (Le and Mikolov, 2014;Kiros et al., 2015;Pagliardini et al., 2018;Schwenk and Douze, 2017;Arora et al., 2019;Zhu et al., 2018). Among them Sent2Vec (Pagliardini et al., 2018) demonstrated the robustness of generated general-purpose sentence embeddings when transferred to a wide range of prediction benchmarks. Sent2Vec is an unsupervised sentence embedding technique allowing composing sentence embeddings using word vectors along with n-gram embeddings.
There are three distinct language-specific categorizations of the Tweets in the shared task datasets. Among the participated tasks, four are based on English Tweets whereas two tasks are based on French and Russian Tweets. For representing the four English language Tweets, we have used 700-dimensional pre-trained 3 Sent2Vec model trained with English Tweets incorporating bi-gram embedding. For non-English Tweets, we have used 1024-dimensional pre-trained Language-Agnostic Sentence Representations (LASER) embeddings 4 (Schwenk and Douze, 2017). The pre-trained LASER model was based on 93 languages and does not need a specification of the input language. The sentence encoder also supports code-switching, i.e. the same sentences can contain words in several different languages. Overall,

Classification Model Selection
We mainly focus on traditional systems to classify the Tweets. In recent years, deep learning models have shown superior performance in classification tasks (Weissenbacher et al., 2019). However, neural models require longer training time (Livni et al., 2014) and due to time constraint, we have utilized traditional baselines from related works (Weissenbacher et al., 2019). The multi-class classification is modeled as an one-vs-one scheme. We have applied Support Vector Machine (SVM) (Cortes and Vapnik, 1995) with radial basis function kernel and tree-based ensemble models, such as, Extra Trees Classifier (Geurts et al., 2006), Random Forest Classifier (Breiman, 2001) for data modeling. The features for models were learned by sentence embedding models as described in Sec. 3.3 and the final model was chosen based on the 10 split k-fold cross-validation on the down-sampled training dataset. For most of the tasks, the SVM model provided better result based on 10-fold cross-validation with mean F1-score evaluation metric, except for T2 (Fr) task where Extra Tree Classifier gave best mean F-1 score. Based on the experiments, for evaluation of the shared tasks, Extra Tree Classifier was used for T2 (Fr), and SVM models were used for all other participated tasks.

Methodology for Span Detection Task
Task 3 can be divided into two sub-tasks: span detection and concept normalization. We have worked on the detection part of the task and the normalization part is left for the future work. The dictionarybased simple traditional approach has been used in information detection task (Egorov et al., 2004). The approach utilizes a carefully constructed dictionary to identify and tag the related entities from the Tweets. For task 3, we have used a dictionary-based search approach to detect the span of adverse effects. A dictionary was generated from the AMIA shared task 5 with 6,649 annotated instances 6 of adverse effects.

Results and Analysis
The official performance evaluation metrics for the tasks are precision (P), recall (R), and F1-score (F1) computed on the positive class (i.e. minority class). Table 2 reports the performance scores of the tasks described in Tab. 1. The rows of the table correspond to the tasks and the columns correspond to the evaluation scores on validation and test datasets. The mean test scores of all the submissions are also reported in the table. The analysis of the results is based on the validation dataset.
Using the methodology described in this paper, from the table we can observe that the recall score is higher compared to the precision score in most cases. As positive labels refer to the minority class, there are a larger number of negative examples that could become false positives (FP). Conversely, there are fewer positive examples that could become false negatives (FN). The disproportion in class distribution can result in the disproportion of the FPs and FNs which could result in high recall and low precision score. As F1-score is a weighted combination of these two matrices, a low precision score can cause an overall low F1-score. Although random under-sampling was used to balance the training dataset, it does not improve the precision score on the validation and test dataset since the validation and test dataset are still imbalanced that reflect the real class distribution.

Conclusion
This paper reports the preliminary exploration of Twitter-based imbalanced data modeling techniques for health applications. We have explored techniques for efficient syntactic and semantic representation of Tweets. As the focus was on imbalanced class distribution, random under-sampling with novel noise reduction technique was utilized to balance the dataset. The traditional margin-based and ensemble tree-based classifiers were used to classify the Tweets.
The directions for future works involve deeper exploration and extensive methodological improvements to enhance the learning performance. For example, the statistical learning theorems establish the number of examples needed to estimate the accuracy of a classifier as a function of its complexity (VCdimension). However, the class imbalance does not enter these formulas anywhere (Juba and Le, 2019). Overall, a detailed exploration of classification models and evaluation metrics can be done to enhance model performance for skewed data distribution.