{'fraction_non_alphanumeric': 0.03694476514553078, 'fraction_numerical': 0.022968949383241174, 'mean_word_length': 4.850693210095983, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Event Detection and Domain Adaptation with Convolutional Neural Networks

We study the event detection problem using convolutional neural networks (CNNs) that overcome the two fundamental limitations of the traditional feature-based approaches to this task: complicated feature engineering for rich feature sets and error propagation from the preceding stages which generate these features. The experimental results show that the CNNs outperform the best reported feature-based systems in the general setting as well as the domain adaptation setting without resorting to extensive external resources.


Introduction
We address the problem of event detection (ED): identifying instances of specified types of events in text. Associated with each event mention is a phrase, the event trigger (most often a single verb or nominalization), which evokes that event. Our task, more precisely stated, involves identifying event triggers and classifying them into specific types. For instance, according to the ACE 2005 annotation guideline 1 , in the sentence "A police officer was killed in New Jersey today", an event detection system should be able to recognize the word "killed" as a trigger for the event "Die". This task is quite challenging, as the same event might appear in the form of various trigger expressions and an expression might represent different events in different contexts. ED is a crucial component in the overall task of event extraction, which also involves event argument discovery.
Recent systems for event extraction have employed either a pipeline architecture with separate classifiers for trigger and argument labeling (Ji and Grishman, 2008;Gupta and Ji, 2009; Patwardhan 1 https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/ english-events- guidelines-v5.4.3.pdf and Rilof, 2009;Liao and Grishman, 2011;Mc-Closky et al., 2011;Huang and Riloff, 2012;Li et al., 2013a) or a joint inference architecture that performs the two subtasks at the same time to benefit from their inter-dependencies (Riedel and Mc-Callum, 2011a;Riedel and McCallum, 2011b;Li et al., 2013b;Venugopal et al., 2014). Both approaches have coped with the ED task by elaborately hand-designing a large set of features (feature engineering) and utilizing the existing supervised natural language processing (NLP) toolkits and resources (i.e name tagger, parsers, gazetteers etc) to extract these features to be fed into statistical classifiers. Although this approach has achieved the top performance (Hong et al., 2011;Li et al., 2013b), it suffers from at least two issues: (i) The choice of features is a manual process and requires linguistic intuition as well as domain expertise, implying additional studies for new application domains and limiting the capacity to quickly adapt to these new domains.
(ii) The supervised NLP toolkits and resources for feature extraction might involve errors (either due to the imperfect nature or the performance loss of the toolkits on new domains (Blitzer et al., 2006;Daumé III, 2007;McClosky et al., 2010)), probably propagated to the final event detector. This paper presents a convolutional neural network (LeCun et al., 1988;Kalchbrenner et al., 2014) for the ED task that automatically learns features from sentences, and minimizes the dependence on supervised toolkits and resources for features, thus alleviating the error propagation and improving the performance for this task. Due to the emerging interest of the NLP community in deep learning recently, CNNs have been studied extensively and applied effectively in various tasks: semantic parsing (Yih et al., 2014), search query retrieval (Shen et al., 2014), semantic matching (Hu et al., 2014), sentence modeling and classification (Kalchbrenner et al., 2014;Kim,  2014), name tagging and semantic role labeling (Collobert et al., 2011), relation classification and extraction (Zeng et al., 2014;Nguyen and Grishman, 2015). However, to the best of our knowledge, this is the first work on event detection via CNNs so far.
First, we evaluate CNNs for ED in the general setting and show that CNNs, though not requiring complicated feature engineering, can still outperform the state-of-the-art feature-based methods extensively relying on the other supervised modules and manual resources for features. Second, we investigate CNNs in a domain adaptation (DA) setting for ED. We demonstrate that CNNs significantly outperform the traditional featurebased methods with respect to generalization performance across domains due to: (i) their capacity to mitigate the error propagation from the preprocessing modules for features, and (ii) the use of word embeddings to induce a more general representation for trigger candidates. We believe that this is also the first research on domain adaptation using CNNs.

Model
We formalize the event detection problem as a multi-class classification problem. Given a sentence, for every token in that sentence, we want to predict if the current token is an event trigger: i.e, does it express some event in the pre-defined event set or not (Li et al., 2013b)? The current token along with its context in the sentence constitute an event trigger candidate or an example in multiclass classification terms. In order to prepare for the CNNs, we limit the context to a fixed window size by trimming longer sentences and padding shorter sentences with a special token when necessary. Let 2w + 1 be the fixed window size, and x = [x −w , x −w+1 , . . . , x 0 , . . . , x w−1 , x w ] be some trigger candidate where the current token is positioned in the middle of the window (token x 0 ). Before entering the CNNs, each token x i is transformed into a real-valued vector by looking up the following embedding tables to capture different characteristics of the token: -Word Embedding Table (initialized by some pre-trained word embeddings): to capture the hidden semantic and syntactic properties of the tokens (Collobert and Weston, 2008;Turian et al., 2010).
-Position Embedding Table: to embed the relative distance i of the token x i to the current token x 0 . In practice, we initialize this table randomly.
-Entity Type Embedding Table: If we further know the entity mentions and their entity types 2 in the sentence, we can also capture this information for each token by looking up the entity type embedding table (initialized randomly) using the entity type associated with each token. We employ the BIO annotation scheme to assign entity type labels to each token in the trigger candidate using the heads of the entity mentions.
For each token x i , the vectors obtained from the three look-ups above are concatenated into a single vector x i to represent the token. As a result, the original event trigger x is transformed into a matrix The matrix representation x is then passed through a convolution layer, a max pooling layer and a softmax at the end to perform classification (like (Kim, 2014;Kalchbrenner et al., 2014)). In the convolution layer, we have a set of feature maps (filters) {f 1 , f 2 , . . . , f n } for the convolution operation. Each feature map f i corresponds to some window size k and can be essentially seen as a weight matrix of size m t × k. Figure 1 illustrates the proposed CNN.
The gradients are computed using backpropagation; regularization is implemented by a dropout (Kim, 2014;Hinton et al., 2012), and training is done via stochastic gradient descent with shuffled mini-batches and the AdaDelta update rule (Zeiler, 2012;Kim, 2014). During the training, we also optimize the weights of the three embedding tables at the same time to reach an effective state (Kim, 2014).

Dataset, Hyperparameters and Resources
As the benefit of multiple window sizes in the convolution layer has been demonstrated in the previous work on sentence modeling (Kalchbrenner et al., 2014;Kim, 2014), in the experiments below, we use window sizes in the set {2, 3, 4, 5} to generate feature maps. We utilize 150 feature maps for each window size in this set. The window size for triggers is set to 31 while the dimensionality of the position embeddings and entity type embeddings is 50 3 .We inherit the values for the other parameters from Kim (2014), i.e, the dropout rate ρ = 0.5, the mini-batch size = 50, the hyperparameter for the l 2 norms = 3. Finally, we employ the pre-trained word embeddings word2vec with 300 dimensions from Mikolov et al. (2013) for initialization.
We evaluate the presented CNN over the ACE 2005 corpus. For comparison purposes, we utilize the same test set with 40 newswire articles (672 sentences), the same development set with 30 other documents (836 sentences) and the same training set with the remaning 529 documents (14,849 sentences) as the previous studies on this dataset (Ji and Grishman, 2008;Liao and Grishman, 2010;Li et al., 2013b). The ACE 2005 corpus has 33 event subtypes that, along with one class "None" for the non-trigger tokens, constitutes a 34-class classification problem.
In order to evaluate the effectiveness of the position embeddings and the entity type embeddings, Table 1 reports the performance of the proposed CNN on the development set when these embeddings are either included or excluded from the systems. With the large margins of performance, it is very clear from the table that the position embeddings are crucial while the entity embeddings are also very useful for CNNs on ED.  For the experiments below, we examine the CNNs in two scenarios: excluding the entity type embeddings (CNN1) and including the entity type embeddings (CNN2). We always use position embeddings in these two scenarios.

Performance Comparison
The state-of-the-art systems for event detection on the ACE 2005 dataset have followed the traditional feature-based approach with rich hand-designed feature sets, and statistical classifiers such as Max-Ent and perceptron for structured prediction in a joint architecture (Hong et al., 2011;Li et al., 2013b). In this section, we compare the proposed CNNs with these state-of-the-art systems on the blind test set. Table 2 presents the overall performance of the systems with gold-standard entity mention and type information 4 .
As we can see from the  with the joint beam search approach using both local and global features (Li et al., 2013b). This is remarkable since CNN1 does not require any external features 5 , in contrast to the other featurebased systems that extensively rely on such external features to perform well. More interestingly, when the entity type information is incorporated into CNN1, we obtain CNN2 that still only needs sentence level information but achieves the stateof-the-art performance for this task (an improvement of 1.5% over the best system with only sentence level information (Li et al., 2013b)). Except for CNN1, all the systems reported in Table 2 employ the gold-standard (perfect) entities mentions and types from manual annotation which might not be available in reality. Table 3 compares the performance of CNN1 and the feature-based systems in a more realistic setting, where entity mentions and types are acquired from an automatic high-performing name tagger and information extraction system (Li et al., 2013b). Note that CNN1 is eligible for this comparison as it does not utilize any external features, thus avoiding usage of the name tagger and the information extraction system to identify entity mentions and types.

Domain Adaptation Experiment
In this section, we aim to further compare the proposed CNNs with the feature-based systems under the domain adaptation setting for event detection.
The ultimate goal of domain adaptation research is to develop techniques taking training 5 External features are the features generated from the supervised NLP modules and manual resources such as parsers, name tagger, entity mention extractors (either automatic or manual), gazetteers etc.

Methods
F Sentence level in Ji and Grishman (2008) 59.7 MaxEnt with local features in Li et al. (2013b) 64.7 Joint beam search with local features in Li et al. (2013b) 63.7 Joint beam search with local and global features in Li et al. (2013b) 65.6 CNN1: CNN without any external features 67.6 data in some source domain and learning models that can work well on target domains. The target domains are supposed to be so dissimilar from the source domain that the learning techniques would suffer from a significant performance loss when trained on the source domain and applied to the target domains. To make it clear, we address the unsupervised DA problem in this section, i.e no training data in the target domains (Blitzer et al., 2006;Plank and Moschitti, 2013). The fundamental reason for the performance loss of the featurebased systems on the target domains is twofold: (i) The behavioral changes of features across domains: As domains differ, some features might be informative in the source domain but become less relevant in the target domains and vice versa.
(ii) The propagated errors of the pre-processing toolkits for lower-level tasks (POS tagging, name tagging, parsing etc) to extract features: These pre-processing toolkits are also known to degrade when shifted to target domains (Blitzer et al., 2006;Daumé III, 2007;McClosky et al., 2010), introducing noisy features into the systems for higher-level tasks in the target domains and eventually impairing the performance of these higherlevel systems on the target domains.
For ED, we postulate that CNNs are more useful than the feature-based approach for DA for two reasons. First, rather than relying on the symbolic and concrete forms (i.e words, types etc) to construct features as the traditional feature-based systems (Ji and Grishman, 2008;Li et al., 2013b) do, CNNs automatically induce their features from word embeddings, the general distributed representation of words that is shared across domains. This helps CNNs mitigate the lexical sparsity, learn more general and effective feature representation for trigger candidates, and thus bridge the gap between domains. Second, as CNNs minimize the reliance on the supervised pre-processing toolkits for features, they can alleviate the error  propagation and be more robust to domain shifts.

Dataset
We also do the experiments in this part over the ACE 2005 dataset but focus more on the difference between domains. The ACE 2005 corpus comes with 6 different domains: broadcast conversation (bc), broadcast news (bn), telephone conversation (cts), newswire (nw), usenet (un) and webblogs (wl). Following the common practice of domain adaptation research on this dataset (Plank and Moschitti, 2013;Nguyen and Grishman, 2014), we use news (the union of bn and nw) as the source domain and bc, cts, wl as three different target domains. We take half of bc as the development set and use the remaining data for testing. We note that the distribution of event subtypes and the vocabularies of the source and target domains are quite different (Plank and Moschitti, 2013). Comparing CNN1 and the baseline systems, we see that CNN1 performs comparably with the baseline systems on the source domain (in-domain performance) (as expected), substantially outperform the baseline systems on two of the three target domains (i.e, bc and cts), and is only less effective than the joint beam search approach on the wl domain; (iii) Finally and most importantly, we consistently achieve the best adaptation performance across all the target domains with CNN2 by only introducing entity type information into CNN1. In fact, CNN2 significantly outperforms the feature-based systems with p < 0.05 and large margins of about 5.0% on the domains bc and cts, clearly confirming our argument in Section 3.3 and testifying to the benefits of CNNs on DA for ED.

Conclusion
We present a CNN for event detection that automatically learns effective feature representations from pre-trained word embeddings, position embeddings as well as entity type embeddings and reduces the error propagation. We conducted experiments to compare the proposed CNN with the state-of-the-art feature-based systems in both the general setting and the domain adaptation setting. The experimental results demonstrate the effectiveness as well as the robustness across domains of the CNN. In the future, our plans include: (i) to explore the joint approaches for event extraction with CNNs; (ii) and to investigate other neural network architectures for information extraction.