{'fraction_non_alphanumeric': 0.03334935967481096, 'fraction_numerical': 0.018663403120765767, 'mean_word_length': 4.928997149520601, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition

This paper presents the results of the two shared tasks associated with W-NUT 2015: (1) a text normalization task with 10 participants; and (2) a named entity tagging task with 8 participants. We outline the task, annotation process and dataset statistics, and provide a high-level overview of the participating systems for each shared task.


Introduction
As part of the 2015 ACL-IJCNLP Workshop on Noisy User-generated Text (W-NUT), we organized two shared tasks: (1) a text normalization task (Section 2); and (2) a named entity tagging task (Section 3).
In the text normalization task, participants were asked to convert non-standard words to their standard forms for English tweets. Participating systems were classified by their use of resources, into a constrained and an unconstrained category: constrained systems were permitted to use only the provided training data and off-the-shelf tools; unconstrained systems, on the other hand, were free to use any public tools and resources. There were 6 official submissions in the constrained category, and 5 official submissions in the unconstrained category. Overall, deep learning methods and methods based on lexicon-augmented conditional random fields (CRFs) achieved the best results. The winning team achieved a precision of 0.9061 precision, recall of 0.7865, and F1 of 0.8421. The named entity recognition task attracted 8 participants. The majority of teams built their systems using linear-chain conditional random fields (Lafferty et al., 2001), and many teams also used brown clusters and word embedding features (Turian et al., 2010). Notable new techniques for named entity recognition in Twitter include a semi-Markov MIRA trained tagger (nrc), an end-to-end neural network using no handengineered features (multimedialab), an approach that weights training data to compensate for concept drift (USFD), and a differential evolution approach to feature selection (iitp). The submission from the winning team (ousia) achieved suprisingly good performance on this difficult task, near the level of inter-rater agreement.

Text Normalization Shared Task
In this section, we outline the Twitter Text Normalization Shared Task, describing the data and annotation process, and outlining the approaches adopted by participants.

Background
Non-standard words are present in many text genres, including advertisements, professional forums, and SMS messages. They can be the cause of reading and understanding problems for humans, and degrade the accuracy of text processing tools (Han et al., 2013;Plank et al., 2014a;Kong et al., 2014). Text normalization aims to transform non-standard words to their canonical forms (Sproat et al., 2001;Han and Baldwin, 2011) as shown in Figure 1. Common examples of non-standard words include abbreviations (e.g., u "you"), and non-standard spellings (e.g., cuming "coming" or 2mr "tomorrow"). The prevalence of non-standard words in social media text results in markedly higher out-of-vocabulary (OOV) rates; normalizing the text brings OOV rates down to more conventional levels and makes the text more amenable to automatic processing with off-theshelf tools which have been trained on edited text.
Text normalization over Twitter data has been addressed at different granularities. For instance, non-standard words can be considered as spelling errors at the character (Liu et al., 2011) or word level (Wang and Ng, 2013). Text normalization can also be approached as a machine translation task, whereby non-standard words are mapped to more canonical expressions (Aw et al., 2006). Other approaches have involved deep learning (Chrupała, 2014), cognitively-inspired approaches (Liu et al., 2012), random walks (Hassan and Menezes, 2013), and supervision using automatically-mined parallel data (Ling et al., 2013).
One major challenge in text normalization research has been the lack of annotated data for training and evaluating methods. As a result, most Twitter text normalization methods have been unsupervised or semi-supervised (Cook and Stevenson, 2009;Han et al., 2012;Yang and Eisenstein, 2013), and evaluated over small-scale handannotated datasets. This has hampered analysis of the strengths and weaknesses of individual methods, and was our motivation in organizing the lexical normalization shared task.

Shared Task Design
This lexical normalization shared task is focused exclusively on English, and was designed with three primary desiderata in mind: (1) to construct a much larger dataset than existing resources; (2) to allow all of 1:1, 1:N and N :1 word n-gramm appings; and (3) to cover not just OOV non-standard words but also non-standard words that happen to coincide in spelling with standard words. In all three regards, the shared task expands upon the scope of the de facto evaluation datasets of Han and Baldwin (2011) and Liu et al. (2011).
One constraint that was placed on candidate tokens for normalization was that they should be all-alphanumeric. For normalization, we adopted American spelling.
In order to establish a more level playing field for participants, but also encourage the use of a wide range of resources, participants were required to nominate their system categories: • Constrained: participants could not use any data other than the provided training data to perform the text normalization task. They were allowed to use pre-trained tools (e.g., Twitter POS taggers), but no normalization lexicons or extra tweet data.
• Unconstrained: participants could use any publicly accessible data or tools to perform the text normalization task.
Evaluation was based on token-level precision, recall and F-score.

Preprocessing
We first collected tweets using the Twitter Streaming API over the period 23-29 May, 2014, and then used langid.py (Lui and Baldwin, 2012) 1 to remove all non-English tweets. Tokenization was performed with CMU-ARK tokeniser. 2 To ensure that tweets had a high likelihood of requiring lexical normalization, we filtered out tweets with less than 2 non-standard words (i.e. words not occurring in our dictionary -see Section 2.2.3). While this biases the sample of tweets, the decision was made at a pragmatic level to ensure a reasonable level of lexical normalization and "annotation density". This was based on a pilot study over a random sample of English tweets, in which we found that many non-standard words were actually unknown named entities which did not require normalization. In all, 5,200 randomlysampled English tweets were annotated for the shared task dataset.

Annotation
12 interns and employees at IBM Research Australia were involved in the data annotation. All annotators had a high level of English proficiency (IELTS 6.0) and were reasonably familiar with Twitter data. Each annotator labeled at least 200 tweets, and each tweet was independently labeled by two annotators based on the annotation guidelines. 3 As part of this, any non-English tweets misclassified by langid.py were manually removed from the dataset. This resulted in the final size of the annotated dataset dropping to 4,917 tweets. All annotations were completed within two weeks, and achieved an average Cohen's  of 0.5854.
For all instances of annotator disagreement, an annotator who was not involved in the first-pass annotation process was asked to adjudicate in the following week. During the course of the shared task, we additionally examined and incorporated a small number of annotation corrections reported by participants.

English Lexicon
It is impossible to reach consensus on the dividing line between standard words and nonstandard words (e.g. are footie, y'all and youse non-standard or standard words?). We artificially arrive at such a dividing line via membership in a prescribed lexicon of English. Specifically, we use the SCOWL database with American spellings as the default English lexicon. 4 The SCOWL database integrates words from multiple sources and also contains valid word spelling variations, which makes it an excellent English lexicon for this shared task. As suggested in the database guidelines, we used a dictionary size of 70%, such that the lexicon contains words found in most dictionaries, but also many high-frequency proper nouns such as Obama and Facebook.
The overall English lexicon (after deduplication) contains 165,458 words. This lexicon was used: (a) to pre-filter data, i.e., tweets with less than two tokens not in this lexicon are dropped from our annotations; and (b) as the basis of the standard words for normalization.

Dataset Statistics
The dataset was randomly split 60:40, into 2,950 tweets for the training data and 1,967 tweets for the test data. Table 1 details the number of (possibly multi-word) tokens in each of the training and 3 http://noisy-text.github.io/files/ annotation_guideline_v1.1.pdf 4 Version 2014.11.17 was used. Training ratio 0.587 0.597 0.500 0.589 Table 1: Numbers of non-standard words in the training and test datasets for the lexical normalization task, broken down into 1:1, 1:N and N :1 mappings from non-standard words to standard words. "Training ratio" represents the number of non-standard words in the training data divided by the overall non-standard words in that category.  test data that were normalized based on a 1:1, 1:N or N :1 mapping. We additionally include the proportion of tokens in each category that were contained in the test data, to confirm that the dataset is relatively balanced in composition between the training and test partitions.
Overall, 373 non-standard word types were found in the intersection of the training and test data. The number of non-standard word types unique to the training and test partitions was 777 and 488, respectively. We further show the top-

Normalization Approaches and Discussion
Overall, 10 teams submitted official runs to the shared task: 6 teams participated in the constrained category, 5 teams in the unconstrained category, and 1 team in both categories. 5 The normalization results for each category are shown in Tables 3 and 4. Overall, common approaches were lexicon-based methods, CRFs, and neural network-based approaches. Among the constrained systems, neural networks achieved strong results, even without off-the-shelf tools. In contrast, CRF-and lexicon-based approaches were shown to be effective in the unconstrained category. Surprisingly, the best overall result was achieved by a constrained system, suggesting that the relative advantage in accessing additional datasets or resources has less impact than the quality of the underlying model that is used to model the task.
NCSU SAS NING (Jin, 2015) Normalization candidates were generated based on the training data, and scored based on Jaccard index over character n-gram[ s]. Candidates were evaluated using random forest classifiers to offset parameter sensitivity, using features including normalization statistics, string similarity and POS.
NCSU SAS WOOKHEE (Min et al., 2015) Word-level edits are predicted based on long-short term memory (LSTM) recurrent neural networks (RNN), using character sequences and POS tags as features. The LSTM is further complemented with a normalization lexicon induced from the training data.
NCSU SAS SAM (Leeman-Munk et al., 2015) Two forward feed neural networks are used to predict: (1) the normalized token given an input token; and (2) whether a word should be normalized or left intact. Normalized tokens are further edited by a "conformer" which down-weights rare words as normalization candidates.

IITP (Akhtar et al., 2015b)
A CRF model is trained over the training data, with features including word sequences, POS tags and morphology features. Post-processing heuristics are used to post-edit the output of the CRF. 5 One team (GIGO) didn't submit a description paper.
DCU-ADAPT (Wagner and Foster, 2015) A generalized perceptron method is used generate word edit operations, with features including character n-gram[ s], character classes, and RNN language model hidden layer activation features. The final normalization word is selected based on the noisy channel model with a character language model.
IHD RD (Supranovich and Patsepnia, 2015) non-standard words are identified using a CRF tagger, using features such as token-level features, contextual tokens, dictionary lookup, and edit distance. Multiple lexicons are combined to generate normalization candidates. A query misspelling correction module (i.e., DidYouMean) is used to post-process the output.
USZEGED (Berend and Tasnádi, 2015) A CRF model is used to identify tokens requiring normalization, and determine the type of normalization required. Normalization candidates are then proposed based on revised edit distance. The final normalization candidate is selected on the basis of n-grams tatistics.
BEKLI (Beckley, 2015) A substitution dictionary is constructed in which keys are non-standard words and values are lists of potential normalizations. Frequent morphology errors are captured by hand-crafted rules. Finally, the Viterbi algorithm is applied to bigram sequences to decode the normalized sentence with maximum probability.

LYSGROUP (Mosquera et al., 2015)
A system originally developed for Spanish text normalization was adapted to English text normalization. The method consists of a cascaded pipeline of several data adaptors and processors, such as a Twitter POS tagger and a spell checker.

Named Entity Recognition over Twitter
The second shared task of WNUT2015 is named entity recognition over Twitter data. Named entity recognition is a crucial component in many information extraction pipelines, however the majority of available NER tools were developed for newswire text and perform poorly on informal text genres such as Twitter. While performance on named entity recognition in newswire is quite high (Tjong Kim Sang and De Meulder, 2003), state-   Table 4: Results of the unconstrained systems for the lexical normalization shared task of-the-art performance on Twitter data lags far behind.
The diverse and noisy style of user-generated content presents serious challenges. For instance tweets, unlike edited newswire text, contain numerous nonstandard spellings, abbreviations, unreliable capitalization, etc.
Another challenge is concept drift (Dredze et al., 2010;Fromreide et al., 2014); the distribution of language and topics on Twitter is constantly shifting leading to degraded performance of NLP tools over time. To evaluate the effect of drift in a realistic scenario, the current evaluation uses a test set from a separate time period, which was not announced to participants until the (unannotated) test data was released at the beginning of the evaluation period.
To address these challenges, there has been an increasing body of work on adapting named entity recognition tools to noisy social media text (Derczynski et al., 2015b;Plank et al., 2014a;Ritter et al., 2011;Plank et al., 2014b), however different research groups have made use of different evaluation setups (e.g. training / test splits) making it challenging to perform direct comparisons across systems. By organizing a shared evaluation we hope to help establish a common evaluation methodology (for at least one dataset) and also promote research and development of NLP tools for user-generated social media text genres.

Training and Development Data
The training and development data for our task was taken from previous work on Twitter NER (Ritter et al., 2011), which distinguishes 10 different named entity types (see Table 5 for the set of types). The data was split into 1,795 annotated tweets for training (train) and 599 as a development set (dev). Participants were allowed to use the development data for training purposes in their final submissions. This data was gathered in September 2010 and annotated by the 5th author.

Test Data Annotation
The test data was randomly sampled from December 2014 through February 2015. Two native English speakers were recruited to independently annotate the test data. The annotators were presented with a set of simple guidelines 6 that cover common ambiguous cases and also instructed to refer to the September 2010 data for reference. The BRAT tool 7 was used for annotation. A screenshot of the interface presented to annotators is shown in Figure 2. During an initial training period, both annotators independently labeled a set of 200 tweets after which disagreements were discussed and resolved before moving on to annotate the final test set. This initial annotation was only done for the purpose of training the annotators and the resulting data was discarded.
The annotators then went on to double-annotate a set of 1,425 messages. An adjudicator, the annotator of the training and dev sets, went through each message and resolved disagreements. The dataset was randomly split into 425 messages as an additional development set (dev2015) which was released to participants at the beginning of the evaluation period. The remaining 1,000 messages (test) were used for the final evaluation; annotations on the test data were withheld from participants until the end of the evaluation period. Table 5 presents precision and recall for each of the 10 categories treating one annotator's labels as gold and the other's as predicted. This exposes the challenging nature of this annotation task and can be viewed as a kind of human upper bound on possible system performance, though we believe the consistency of the final annotations to be somewhat higher due to the second pass made by the adjudicator. The value of Cohen's  as measured on word-level annotations is 0.607.
A baseline system was provided to participants which takes a simple approach based on CRFsuite 8 using a standard set of features which include contextual, orthographic and gazetteers generated from Freebase (Bollacker et al., 2008). The evaluation consisted of 2 sub-tasks: one in which participants' systems were required to segment and classify 10 named entity types and one where the task is only to predict entity segmentation (no types).

Approaches
Eight teams (Table 6) participated in the named entity recognition shared task. A wide variety of approaches were taken to tackle this task. Table 7 summarizes the features used by each team and the machine learning approach taken. Many teams made use of word embeddings and Brown clusters as features. One team (multimedialab) used absolutely no hand-engineered features, relying entirely on word embeddings and a feed-forward neural-network (FFNN) architecture (Godin et al., 2015). Other new approaches to Twitter NER include a semi-Markov MIRA trained tagger developed by the NRC team  and the use of entity-linking based features by ou-   Table 6: Team ID and affiliation of the named entity recognition shared task participants.
sia (Yamada et al., 2015). All the other teams used CRFs. On top of a CRF, the iitp team used a differential evolution based technique to obtain an optimal feature set. Most systems used the training data as well as both dev sets provided to train their system, except multimedialab which did not use dev2015 as training data and NRC which only used train.

9
Tables 8 and 9 report the results obtained by each team for segmentation and classification of the 10 named entity types and for segmentation only, respectively.

System Descriptions
Following is a brief description of the approach taken by each team: 9 A post-competition analysis of the effect of training on development sets is presented in the NRC system description paper .    multimedialab (Godin et al., 2015) The goal of the multimedia lab system was to only use neural networks and word embeddings to show the power of automatic feature learning and semi-supervised methods. A Feed-Forward Neural Network was first trained, that used only word2vec word embeddings as input. Word embeddings were trained on 400 million unlabeled tweets. Leaky ReLUs were used as activation function in combination with dropout to prevent overfitting. A context window of 5 words was used As input (2 words left and right). The output is a single tag of the middle word. Afterwards, a rule-based post-processing step was executed to ensure every I-tag has a B-tag in front of it and that all tags within a single span are of the same type. Train and dev were used as training data and used dev 2015 as validation set.
NLANGP (Toh et al., 2015) The NLANGP team modeled the problem as a sequential labeling task and used Conditional Random Fields. Several post-processing steps (e.g. rulebased matching) were applied to refine the system output. Besides Brown clusters, Kmeans clusters were also used; the K-means clusters were generated based on word embeddings.
nrc  NRC applied a MIRAtrained semi-Markov tagger with Gazetteer, Brown cluster and Word Embedding features. The Word Embeddings were built over phrases using Word2Vec's phrase finder tool, and were modified using an auto-encoder to be predictive of Gazetteer membership.
ousia (Yamada et al., 2015) The main characteristics of the ousia method is enhancing the performance of Twitter named entity recognition using entity linking. Once entity mentions are disambiguated to the knowledge base entries, high-quality knowledge can be easily extracted from a knowledge base such as the popularity of the entity, the classes of the entity, and the likelihood that the entity appears in the given context. They adopted supervised machine-learning with features including the results of NER and various information of the entity in knowledge bases. We use Stanford NER was used for the NER and in-house end-to-end entity linking software was applied for entity linking.
USFD (Derczynski et al., 2015a) Feature extraction was based on large Brown clusters, gazetteers tuned to the input data, and distant supervision from Freebase. The representation was tuned for drift by down-weighting temporally distant training examples. The classifier was a linear chain CRF with hyperparameters tuned for Twitter.

Summary
In this paper, we presented two shared tasks on Twitter text processing: Lexical Normalization and Named Entity Recognition. We detailed the task setup and datasets used in the respective shared tasks, and also outlined the approach taken by the participating systems. Both shared tasks were of a scale substantially larger than what had previously been attempted in the literature, with two primary benefits. First, we are able to draw stronger conclusions about the true potential of different approaches. Second, through analyzing the results of the participating systems, we are able to suggest potential research directions for both future shared tasks and noisy text processing in general.