{'fraction_non_alphanumeric': 0.031571722717913524, 'fraction_numerical': 0.016643788606726148, 'mean_word_length': 4.720314033366045, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transformer Models for Drug Adverse Effects Detection from Tweets

In this paper we present the drug adverse effects detection system developed during our participation in the Social Media Mining for Health Applications Shared Task 2020. We experimented with transfer learning approach for English and Russian, BERT and RoBERTa architectures and several strategies for regression head composition. Our final submissions in both languages overcome average F1 by several percents margin.


Introduction
In the world of pandemic threats it is important to pay a special attention to the process of drug discovery. For the pharmaceutical industry it was always important not only to develop a new drug, but also to detect its possible Adverse Effects (AEs) even to smallest group populations as soon as possible. Although early stages of a clinical trial reveal most of a drug AEs, constant monitoring and feedback collection at the last pharmacovigilance stage is mandatory as it allows to identify rare or slowly evolving AEs. With current development of social media networks and artificial intelligence methods it is tempting to mine this information from the publicly available user data such as Twitter messages. If successful one can get an additional source of valuable information.
The above stated problem got into the research focus of 2020 Social Media Mining for Health Applications (SMM4H) Workshop (Klein et al., 2020). The organizers divided this problem into three consequent sub-tasks: 1) medical tweets detection; 2) classification of tweets that report AEs and 3) extraction and normalization of AEs from text. Specifically the second task was to build a binary classification model able to distinguish tweets that report an AE of a medication from those that do not. We concentrate our efforts only on this task as it is the only one that offers multilingual version (English, French and Russian).
In the recent years, methods for Natural Language Processing (NLP) developed rapidly. Nowadays transformer-based neural architectures (Vaswani et al., 2017) dominate the field. Our goal for this study was to benchmark BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) architectures and several strategies for regression head composition on this practical problem from medical domain with realworld data.

Data
The organizers provided labeled data alone with the Train / Validation split. Table 1 summarize statistics about the data for our languages of interest. As one can see the subset of the data for Russian language is very limited which affects the results (see Section 4).
In our experiments we don't use the provided split, but join train and validation parts for each language and perform 5-fold Cross-Validation (CV) procedure (Bishop, 2006). This allows us to estimate standard deviations for the models.   The overall tendency of recent deep learning models is to avoid intermediate tasks and preprocessing, so the only preprocessing steps we perform with the labeled data is to remove user names and drop duplicate tweet messages (about 3.5% for the English).

Our approach
As the train data were very limited we resort to a transfer learning approach. That is, we take an NLP model pre-trained on a large corpus of texts (a backbone model) and fine-tune it for a specific task at hand. In this study as such backbone models for English we used BERT-base and RoBERTa-base from Transformers library (Wolf et al., 2019). For the Russian BERT model we fine-tuned RuBERT (Kuratov and Arkhipov, 2019). There was no Russian RoBERTa model, so we trained one from scratch on the public data (Shavrina and Shapovalova, 2017) and made it available. 1 This model was trained with Adam optimizer (starting learning rate 3 × 10 −6 ) on 16 GPU (Tesla V100) and the batch size of 1024 (with gradient accumulation) for about 60K steps.
Originally BERT and RoBERTa architectures designed a special classification token (cls) to summarize the state of a whole input text sequence (into a single embedding) and allow to configure downstream tasks (thought classification or regression head). It is known that in NLP-related tasks max-pooling operation over the embeddings associated with sentence keyword selection and the mean-pooling is a way of encoding the general meaning of the sentence. So we experimented with this pooling strategies over the last encoder states. Additionally the option with concatenation of embeddings (cls & max & mean) were explored. The resulting hidden state followed by additional fully connected layer of the same size and ends with regression output. Based on CV metric maximization strategy the threshold were selected to make final binarization of predictions.

Results and Conclusions
The primary evaluation metric for the task was F 1 -measure toward the positive class as a trade-off between Precision (P ) and Recall (R) (Manning et al., 2008). The 5-fold CV results of our experiments are shown in  Table 2: 5-fold CV results for language, architecture and specific regression head (F 1 ± F std , %).
First of all, Table 2 shows that the English model performs much better with respect to the Russian one, probably because of the training data size (large standard deviation is in favor of this hypothesis). Second, the right choice of a regression head do help to get better performance.
For the Russian submission we selected BERT model with mean regression head. In this case our system metrics is perfectly matched between CV and test, F 1 = 48% (P = 36.1%, R = 70.5%). Which is also 5.3% improvement compared to average scores for this task F 1 = 42.7% (P = 36.2%, R = 58.3%).
The results allow us to conclude that the drug adverse effects detection in raw text still remains challenging problem and requires more elaborate methods to reach the acceptable quality.