{'fraction_non_alphanumeric': 0.028632230605532986, 'fraction_numerical': 0.01020506867866125, 'mean_word_length': 4.693006607929515, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Tale of tails using rule augmented sequence labeling for event extraction

The problem of event extraction is a relatively difficult task for low resource languages due to the non-availability of sufficient annotated data. Moreover, the task becomes complex for tail (rarely occurring) labels wherein extremely less data is available. In this paper, we present a new dataset (InDEE-2019) in the disaster domain for multiple Indic languages, collected from news websites. Using this dataset, we evaluate several rule-based mechanisms to augment deep learning based models. We formulate our problem of event extraction as a sequence labeling task and perform extensive experiments to study and understand the effectiveness of different approaches. We further show that tail labels can be easily incorporated by creating new rules without the requirement of large annotated data.


Introduction
Event occurrences involve several entities such as time, date, place, reason, etc. Event extraction recovers structured representations from the text, often characterised by complex argument and nested events, involving several entities. Entities have associated properties and attributes such as reason of the happening, after-effects of the events, etc. Event extraction has several applications in tasks that include text summarization, knowledge-base construction, machine translation, etc.
Entity extraction is a pre-requisite for building any event extractor. Additionally, event extraction (EE) often requires the discovery of relations between entities and dependencies between relations from the text. Typically, EE systems find triggers and their associated arguments. The discovery of triggers and arguments poses several challenges. Firstly, event detection is highly contextual driven.
The same event may appear with different trigger words and a trigger word can evoke different event expressions. For example, consider a sentence 'On 29 December 2017 a massive fire broke in Kamala Mills, Mumbai the capital of Maharashtra, killed at least 14 people and injured several'.
The trigger word killed may co-occur with fire accident or attack or any other keyword associated with an incident involving such a trigger word. Similarly, the trigger word killed can evoke different events depending upon the context. Secondly, the presence of a large number of entities requires large annotated data focused on deep learning based systems. Typical deep learning systems require a sufficiently large amount of data for training (Sun et al., 2017). Named entity recognition (NER) systems are not effective for extracting a large number of tags given less data. Third, of particular concerns are the labels with very few instances in the dataset, hereafter referred to as 'tail labels'. Most event extraction methods are not designed to be fair to tail labels and land up being biased by the more frequent labels. Additionally, the absence of embeddings that capture language models effectively is especially pronounced for low resource languages. This can lead to propagation of errors resulting from the improper initialisation of embeddings for out-ofvocabulary words.
In this paper, we address these challenges to create an end-to-end EE system in the disaster domain in 5 languages, namely, Marathi, English, Hindi, Bengali and Tamil. We introduce rule augmented deep learning methods and demonstrate the effectiveness of our approach with extensive experiments. We model EE as a sequence labelling task and investigate ways of enhancing state-of-the-art entity extractors by augmenting using rule-based approaches. Our contributions are summarised as follows : • We re-purpose existing rule augmented deep learning models for learning event structures that captures event arguments and their interdependencies on disaster domain in lowresource languages.
• We conduct extensive experiments on our dataset and demonstrate the importance of rule-augmented deep learning models in improving performance on tail labels.
• We release a new dataset named InDEE-2019 that consists of tagged event extraction data in the disaster domain covering five languages: Marathi, English, Hindi, Bengali and Tamil.

Related Work
Event extraction is a well studied problem, albeit mostly in the English language. Presently, most of the event extraction method consist of deep learning model which requires large annotated training datasets. These data hungry methods work in resources rich language like English (Nguyen et al., 2016) but it fails in low resource languages like Indic languages. Few previous works have used rule based (Valenzuela-Escárcega et al., 2015), however, the model is not able to learn complex functions of data. Other work (Reschke et al., 2014) uses sources of external information from the knowledge bases to improve the performance of linear chain Conditional random field (CRF) baselines. Therefore, we need hybrid approach to leverage benefits of both methods. Snorkel (Ratner et al., 2017) deals with candidate extractors and use weak supervision sources to classify the correct set of arguments. However, snorkel requires candidate to be NER tagged and it works well with lesser number of labels. To address this issue, we have augmented rules in different ways with deep learning architectures to learn contextual information along with rules.

Event Extraction Task
We focus on the EE task and model as a sequence labeling problem. We use the following terminologies throughout the paper: • Event Trigger : The main word that identifies occurrence of the event mentioned in the document.
• Event Arguments : Several words that define an events such as place , time , reason , after-effects , participant , casualties.
We are interested in extracting event triggers, event arguments from the document. Table 1 shows the number of labels for each language.

Approach
In this section, we describe our approach for the event extraction task. We model the extraction as a sequence labeling problem. We adopt Bidirectional LSTM (Long Short Term Memory) and CRF (Conditional Random Fields) (Huang et al., 2015) for sequence labeling as a baseline and augment it by incorporating information from rules.

Bi-directional LSTM
Recurrent Neural Networks (RNN) has achieved significant gains in sequence labeling tasks. Due to its sequential architecture, RNNs are able to capture previous sequential inputs to predict the output. LSTM are variants of RNN and has outperformed RNNs to capture long-range dependencies in the sequence (Hochreiter and Schmidhuber, 1997). Recently, state-of-the-art named entity extractors (Lample et al., 2016) have used bi-directional LSTMs with CRF. CRF effectively capture transition states during inference, however, they do not perform well on skewed distribution of tags . CRF layer enables to add constraints during inference such that invalid label sequences are discarded from the search space. However, this benefit diminishes on less training data when transition probabilities are not learned effectively. Given an input sequence words w 1 , w 2 , . . . w n , LSTM capture current state α i based on current input word w i and previous hidden state α i−1 . Single directional LSTM are not sufficient to capture dependencies from future words. In order to address this issue, a LSTM in reverse direction is used to generate another hidden vector representation β i−1 .
It is well documented that LSTMs reduces vanishing gradient problem to model interdependencies in long sequences. LSTM output tag probability distribution for each individual word in an input sequence while CRF gives a score for the tag sequence. Due to skewed tag distribution in our dataset, CRF gives very low probabilities for the rarely occurring tags. Therefore, we def positive_dictionary(current_word,word_vicinity): def negative_dictionary(current_word,sentence): Similarity with words(synonyms of flood): Figure 1: The current word is checked for the similarity with positive and the negative dictionary in marathi along with the vicinity words as shown in dotted box.
chose Bi-LSTM as our baseline model and eliminate CRF due to highly skewed tag distribution in our dataset.

Rule Definition
Our approach extends Bi-LSTM with a rule-based approach that generates a rule vector for each word. The rule vectors are built using event anchors to capture class information. We create several dictionaries that correspond to characteristics of event triggers. These dictionaries augment the features learned by the Bi-LSTM. Figure 1 gives an example of dictionary for developing rules.

Synonym-based dictionary
The dictionary corresponds to synonyms of the trigger word. We create sets of words that correspond to the synonyms of the trigger word for each language. For instance, synonyms of flood are shown in Figure 1. Once we create robust dictionary for one language is easy to extend it to other languages. We will refer synonym dictionaries as positive dictionaries.

Negative Dictionary
In several documents, many sentences refer to events that happened in the past or have a chance of happening. Such events needs to be ignored by our system such that unreasonable mentions of events are not considered as actual events. For example, ' There exists a strong possibility of spreading of Malaria after 2015 floods in Mumbai '. The possibility word in the sentence , causes the event to classify as probable event. Therefore, it is tagged as a negative mention in the annotation. In order to capture these negative instances, we created a negative dictionary as shown in Figure  1 that contains such words.

Rule Vector
Given a sentence s, containing word sequences w 1 , w 2 , . . . , w n , we create rule vector r i for each word w i . Due to overlapping labels in our tag set, r i is a multi-hot vector. For instance, any attack can belong to both Normal bombing and Terrorist attack.
Algorithm 1 generates rule vector r i for each word w i in the sentence s. Dimension of r i is equal to number of labels + 1 (for other tag). If any word w i in the sentence s found in the negative dictionary, neg then the word should be tagged as other. If any word in window of {w i − l, w i + l} for current word w i , present in particular dictionary syn t then w i should be tagged as tag t. If none of the word from the window matched with any of the synonym dictionaries then current word w i should be tagged as other. The algorithm for rule vector formation is shown in Algorithm 1. We have taken multiple values of l for experimentation. We also used similarity based matching in instead of exact word matching.

Techniques for incorporating rule vector
We use rules to augment our deep learning labeling architecture using following three methods. The complete architecture is shown in Figure 2.

Augment embeddings with rule vector
In this method, we append the rule vector along with the word representation. In the encoding phase, we extract word representation e i for word w i using pre-trained fastText embeddings (Bojanowski et al., 2017) fine-tuned on our dataset. The word embeddings are appended with the rule vectors r i . The concatenated vector on application of dropouts (Srivastava et al., 2014) is fed to the Bi-LSTM that extract essential features and learn representation.

Algorithm 1 Algorithm for creating rule vector
Require: Sentence s with words w 1 , w 2 , . . . , w n , current word w i in s, synonym dictionary syn t for each label t and negative dictionary neg, 2l is the window size for w i , f lag indicates word appeared in syn or not.

Explicit rule addition
In addition to the word embeddings fed to the Bi-LSTM, a rule vector through a separate Bi-LSTM is passed. Two parallel Bi-LSTM are fed with word embeddings and rule vectors. The hidden layer representation are concatenated to learn a joint representation of words and rules. The architecture is shown in Figure 2.
We expect this method to retain essential information from rules. In comparison to the concatenation approach proposed in the previous section, we expect it to retain more relevant information by passing rules explicitly.

Rule projection using distillation
In this method, the knowledge of rules is distilled within deep learning network. This is achieved by biasing the weights of neural networks according to the rule vector. (Hu et al., 2016) proposed a student and teacher model such that model weights are learnt within constraints of rule based system. The teacher-student models bias each other such that weights are learnt by modelling logical rules as constraints and projecting rules within constrained space. Using similar framework we made small changes to make it work to with side information provided by the rules. As given by the eq. (1) The teacher distribution, q * (T |w i ) is the modified form of the student distribution, p(T, w i ) to fit to transform according to rule vector r i keeping the basic intuition same, where T is the tag distribution. The formulation modifies the prediction probabilities to have a bias towards the rule vector. For example, r i being a multi-hot vector can be interpreted as set of labels(tags) assigned by the rules. The formula tries to the bias the student distribution p θ (T |w i ) by keeping the probability of the tag predicted by the rules, and penalises the probabilities of other labels by a factor of exp (C), where C is an arbitrary constant, we set C = 1 for our experiments . Using eq.(2), the student tries to mimic the teacher prediction s (t) n with an imitation parameter π and also tries to optimize for ground truth T n . For our experiments we set our imitation parameter to value 0.4.
Parameters of the student distribution θ is updated using the above update rule, σ θ (w n ) is the student prediction at the n-th word in a sequence where l is application specific loss (cross-entropy here), KL is KL divergence loss. Final inference is made using the projected distribution, i.e. the teacher distribution.

Word Embeddings
Due to the very nature of low resource languages, out-of-vocabulary words are common occurrences. To handle such words, we use pretrained fastText word embeddings (Bojanowski et al., 2017) and fine-tune on our corpus. We observe that word coverage of fast-text embeddings on Marathi, Tamil and Bengali is around 50%.
Here, we focused only on building an EE system instead of developing word embeddings.

Dataset
We release the new dataset named InDEE-2019 which consists of tagged event extraction data in disaster domain covering five languages: Marathi, English, Hindi, Bengali and Tamil. Dataset statistics can be found in Table 1.
For each language, we crawled disaster related documents from regional news websites. To annotate this documents we followed IOB (Inside-Other-Beginning) tagging scheme proposed by (Ramshaw and Marcus, 1999). IOB tagging helps in differentiating between starting and ending of adjacent tags. In our disaster related documents, adjacent occurrence of same tags forms a phrase. Therefore, we adopted a simpler scheme that merges B and I together and tag under TO (T:Tag and O:Other) scheme. Table 2 represents sample sentence from the tagged dataset where first column contains tokens of the sentence, second column represents doc id corresponding to sentence and third column represents tag of token. TO tagging scheme can be seen in third column.
We hired linguistic experts for each language to annotate the dataset as per tagging guidelines. For each language, we divided dataset into three parts train (70%), validation (10%) and test (20%). The train dataset is used to train the model for event extraction task, validation is used for hyperparameter tuning and test dataset is used for testing our model. This dataset is quite challenging because of multiple reasons. First, there are very large number of tags and very less training data. Moreover, there are closely related tags that made the annotation task difficult. Secondly, data distribution is skewed, therefore very less training data exists for many tags. Third, our main focus is on low resource Indic languages that makes the task more challenging.

Results and Discussion
We use Bi-LSTM as our baseline and compare with the proposed three approaches. We conduct experiments on our InDEE dataset on 5 languages, namely, Hindi, Marathi, Bengali, Tamil and English. Our evaluation metric is standard micro-F1 and macro-F1 scores. Micro-F1 score counts the global true positives, false positives and false negatives whereas Macro-F1 captures the average unweighted class scores. Macro does not take class imbalance into consideration. We observe that due to highly skewed label distribution in our dataset, micro score is of more interest to us. For the ease of use, we will call our baseline as A and three proposed approaches as B(4.4.1), C(4.4.2), D(4.4.3).
We train our marathi model on 15.9K sentences and predict on 43 labels. Training sets and sizes for all languages are shown in Table 1. We train our model on varying training set size, namely, 20%, 40%, 60%, 80% and 100% to ascertain the impact of rules with decreasing amount of dataset. Figure 3 shows F1-macro and F1-micro scores for the marathi EE. It is observed that at lesser training instances, our rule based approach is able to outperform baseline on 3K(20%) and 6K(40%) training instances. Both macro and micro F1 scores show improvement over baseline model. As training instances increases, deep learning models are able to learn large number of parameters that were difficult to learn on lesser training instances.

Language
20% 40% B C D B C D Mr 10 8 8 12 8 8 Hi 9 12 8 10 6 6 En 5 5 4 5 2 3 Ta 4 3 4 9 5 6 Bn 8 6 6 8 7 3   Table 6 shows macro-F1 and micro-F1 scores for all 5 languages. We observe that on smaller a) b) Figure 4: a) Comparison of proposed rule based approaches on improvement over all labels. b) Comparison of proposed approaches over tail labels. Lowest stack represents number of labels shown improvement over baseline, middle stack represent count of labels that has equal score with baseline and upper stack represent count of labels that has lesser score than baseline.
training instances, implicit rule approach tend to perform better on Marathi & Hindi. Most of the cases shows that our methods performs better than the baseline on less training data. Due to better word representation and data annotation on English, the parameters are even learnt on smaller dataset by Bi-LSTM. However, for low resource languages, Bi-LSTM is not able to learn parameters on similar training instances. Tail Labels Most deep learning based methods are not able to capture tail labels due to lesser training data. However, this is of interest to us since most real world data has small size and has large label set. Figure 4 shows improvement of F1-score for each classes over baselines. We only considered scores that are greater than baseline scores. We observe that more classes are improved by including rules over baselines. At lesser training instances, we observe that large number of tags are correctly classified. Table 4 shows number of tags improved over baseline. In order to further prove effectiveness of rule based approach, we tested the improvements of classes over tail labels. We chose those tails labels whose sum forms 5% of total training set instances. We notice significant improvement of tags over 20% and 40% training instances. We can see the number of tail labels improved over baseline in table 3. Moreover, we calculated micro-F1 scores for the tails labels. The results are shown in Table 5. We notice significant improvement on tails for 20% and 40% training instances. It is clear from the results that at lesser training instances, our approach is effec-tive in capturing tail labels. While working with low resource languages and lesser annotated data, combination of rules and Bi-LSTM gives higher score. Additionally, rule based approach is effective at capturing tail labels.

Conclusion
In this paper, we introduce a hybrid approach for automatic extraction of events and arguments. We present a new dataset in the disaster domain for five languages consisting of large number of tags than usual datasets. We propose several variants of rule based system to augment deep learning based models. Extensive experimental results demonstrate that our rule augmented methods outperforms deep learning based models on lesser annotated data and low resource languages. We further shows more improvement on tail labels using our approach. For future work, we plan to integrate cross linking between events and its arguments.