{'fraction_non_alphanumeric': 0.035776201733648545, 'fraction_numerical': 0.014152876280535855, 'mean_word_length': 4.857828655834564, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

Recent trends in NLP research have raised an interest in linguistic code-switching (CS); modern approaches have been proposed to solve a wide range of NLP tasks on multiple language pairs. Unfortunately, these proposed methods are hardly generalizable to different code-switched languages. In addition, it is unclear whether a model architecture is applicable for a different task while still being compatible with the code-switching setting. This is mainly because of the lack of a centralized benchmark and the sparse corpora that researchers employ based on their specific needs and interests. To facilitate research in this direction, we propose a centralized benchmark for Linguistic Code-switching Evaluation (LinCE) that combines eleven corpora covering four different code-switched language pairs (i.e., Spanish-English, Nepali-English, Hindi-English, and Modern Standard Arabic-Egyptian Arabic) and four tasks (i.e., language identification, named entity recognition, part-of-speech tagging, and sentiment analysis). As part of the benchmark centralization effort, we provide an online platform where researchers can submit their results while comparing with others in real-time. In addition, we provide the scores of different popular models, including LSTM, ELMo, and multilingual BERT so that the NLP community can compare against state-of-the-art systems. LinCE is a continuous effort, and we will expand it with more low-resource languages and tasks.


Introduction
Linguistic code-switching 1 (CS) is the multilingual phenomenon that happens when speakers alternate languages within the same sentence or utterance. During the last decade, the CS phenomenon has attracted more research interest from the NLP community. Many researchers have proposed novel methods to handle code-switched data, showing improvements on core NLP tasks such as language identification (LID), named entity recognition (NER), and part-of-speech (POS) tagging. However, many of these approaches are usually evaluated on a few language pairs and a specific domain, and it is not clear whether these models are exclusive to such scenarios or they can generalize to other tasks, domains, and language pairs. Moreover, research in code-switching currently has a slow process of comparison in which researchers have to replicate previous methods to report scores on different datasets. Furthermore, choosing the best-published model for benchmarking purposes is not an easy task either. These problems exist mainly because 1) there is no official benchmark for general code-switching evaluation that allows direct comparisons across multiple tasks, and 2) methods are usually not comprehensively evaluated across datasets with different language pairs. To overcome these problems, we propose a centralized Linguistic Code-switching Evaluation (LinCE) benchmark. We have consolidated a benchmark from preexisting corpora considering the following aspects: 1) multiple language pairs from high-and low-resource languages with a reasonable range of code-mixing indexes (CMI) (Gambäck and Das, 2014), 2) typologically-diverse languages 2 , 3) a variety of NLP tasks including core tasks and downstream 1 We use code-switching and code-mixing equivalently. 2 We also consider the geolocation of such languages to account for places across the world.
applications, and 4) different code-switching domains from social media platforms. LinCE is comprised of four LID datasets, two POS tagging datasets, three NER datasets, and one sentiment analysis (SA) dataset, providing a total of ten datasets (see Table 1). Furthermore, an important contribution of LinCE is the new stratification process to provide fair and, in some cases, official splits for the tasks at hand. This required a careful inspection of the original datasets from which we list five major issues (see Section 4.5) and propose new splits for nine out of the ten datasets. In addition to the LinCE benchmark, we also provide strong baselines using popular models such as LSTMs (Hochreiter and Schmidhuber, 1997), ELMo , and multilingual BERT (Devlin et al., 2018). In our analysis, we evaluate the results of the best model and describe the outstanding challenges in this benchmark. Moreover, LinCE is publicly available at ritual.uh.edu/lince, and we anticipate this benchmark to continue to grow and include new tasks and language pairs as they become available. We hope that LinCE motivates future work and accelerates the progress on NLP for code-switched languages.
In the case of language identification (LID) at the token level, researchers have evaluated approaches such as conditional random fields (CRF) with hand-crafted features (Al-Badrashiny and Diab, 2016), LSTM models with word and character embeddings (Mave et al., 2018;Samih et al., 2016), code-mixed word embeddings (Pratapa et al., 2018), and transfer learning (Aguilar and Solorio, 2019). While most of these approaches reach over 90% of accuracy regardless of the language pairs, it is hard to determine which model is the best overall and what the trade-offs are by using one instead of the others. Likewise, for part-ofspeech (POS) tagging, the community has explored tools that heavily rely on monolingual hand-crafted linguistic information and morphological features (AlGhamdi et al., 2019), traditional ML techniques (e.g., SVM) with heuristics that exploit monolingual resources (Solorio and Liu, 2008), combined monolingual taggers including CRF and Random Forest (Jamatia et al., 2015), and jointly modeling POS tagging with LID using recurrent neural networks . Although such approaches are effective on their datasets at hand, they are languagespecific and not easy to compare across each other. A slightly different trend has been marked in named entity recognition (NER). Although the main problem in NER has been the lack of datasets, it is until recently that researchers have provided a few corpora on Hindi-English (Singh et al., 2018b), Spanish-English and Modern Standard Arabic-Egyptian Arabic (Aguilar et al., 2018). The participants of the 2018 CALCS competition proposed models based on standard neural NER architectures (e.g., character CNN, followed by a word-based LSTM, and CRF) (Geetha et al., 2018), including variations with attention (Wang et al., 2018b) and multi-task learning (Trivedi et al., 2018). Additionally, most of the participants exploited publicly available resources such as gazetteers as well as monolingual and multilingual embeddings (Winata et al., 2018). While the CALCS competition provided datasets on Spanish-English and Modern Standard Arabic-Egyptian Arabic simultaneously, the participants were allowed to provide predictions on one or both competitions. This flexibility left the question open regarding which model was overall the best across language pairs. Sentiment analysis (SA) on code-switched data has not been explored extensively either. Vilares et al. (2015) provided a Spanish-English polarity annotations for a small section of the 2014 CALCS LID corpus. Their focus was to compare different bag-of-words features using L2regularized logistic regression models often employed in monolingual SA. Concurrent to our work, SemEval-2020 is hosting the first competition for SA in code-switched data, Task 9: Sentiment Analysis for Code-Mixed Social Media Text (Patwa et al., 2020), which covers Spanish-English and Hindi-English with tweets annotated with both sentiment and language identification labels. We adopt the Spanish-English corpus as part of the LinCE benchmark. Although there has been progress in code-switching overall, CS is still lacking advancements in many NLP tasks. Additionally, CS tends to advance guided by languagespecific challenges, usually providing sparse technologies that may not necessarily be effective for other language pairs. By gathering different language pairs and tasks into a single benchmark, we expect LinCE to strive for consolidated and steady progress in code-switching research.

Linguistic Challenges
Although code-switching can happen in more than two languages, this benchmark focuses on language pairs only. The frequent alternations between two languages is precisely what makes the automated processing of codeswitching data difficult. We quantify such complexity using the CMI index proposed by Gambäck and Das (2014) as shown in Table 2. The higher the CMI index, the more alternations the dataset contains, and hence, the more complex the code-switching behavior is. In addition to the alternation of languages, we briefly describe other linguistic challenges that each specific language pair poses to current NLP systems: • Spanish-English (SPA-ENG). While English is a Germanic language, a significant number of words from its current vocabulary have been borrowed from Latin and French since the Middle Ages (Tristram, 1999). This particular set of words tends to overlap with words from Spanish, a Latin-based language. This overlap increases ambiguity and directly affects systems that rely on character-based approaches, for example, in the case of language identification. Codeswitching also appears within the words, often inflecting words by conjugating English verbs using Spanish grammatical rules. This behavior is known as Spanglish (Rothman and Rell, 2007), and it particularly affects non-contextualized word embeddings as it increases the out-of-vocabulary (OOV) rate.
• Hindi-English (HIN-ENG). One of the most challenging aspects of this language pair is the lack of a standardized transliteration system. Speakers transliterate Hindi employing mostly ad-hoc phonological rules to use the English alphabet when writing. Using the same roman alphabet makes code-switching more convenient but the lack of an official standard for transliteration makes it difficult to process with existing resources exclusively available for Hindi with  The CMI scores and the number of tokens across corpora. All Posts describes the number of posts in the corpora and All CMI is the corresponding CMI scores for such samples. Similarly, CS Posts denotes the number of code-switched posts (excluding monolingual posts) and CS CMI is the corresponding CMI scores for such samples. We also show the number of tokens that belong to the language pairs (Lang1, Lang2) as well as the overall number of tokens (All Tokens), which includes other LID labels beyond the language pairs. English is the Lang1 class for English-paired languages; for MSA-EA, Modern Standard Arabic is the Lang1 class. We omit the CMI information for the MSA-EA NER corpus because the corpus does not come with language identification labels.
the Devanagari script. Furthermore, although Hindi loosely follows the subject-object-verb (SOV) structure, its flexible word order poses an additional challenge to NLP systems.
• Nepali-English (NEP-ENG). Similar to HIN-ENG, Nepali is transliterated using the English alphabet when code-switched with English. This behavior makes Nepali speakers to write driven by arbitrary phonological rules that allow the romanization of Nepali using the English alphabet, which excludes the few monolingual resources available for Nepali. Also, Nepali is a subject-object-verb (SOV) language while English is subject-verb-object (SVO). This grammatical difference intuitively encourages more codeswitching points since it is proven that, when codeswitching occurs, the languages involved still preserve their grammatical structure (Solorio and Liu, 2008), which forces more fine-grained alternations to obey the SOV and SVO structures. In practice, we see a large code-switching rate for Nepali-English captured by the averaged CMI index in Table 2, being one of the largest scores while having a corpus of middle size.
• Modern Standard Arabic-Egyptian Arabic (MSA-EA). Arabic is well known for its diglossia (Ferguson, 1959); which combines a number of Arabic dialects with Modern Standard Arabic within the same community. This combination of dialects enables a large occurrence of linguistic code-switching. One of the main challenges with this language pair is that there is a significantly large word overlap while the word meanings can vary depending on the language. Even more, Arabic is a morphologically rich language and it allows multiple word orders, which increases the semantic complexity for NLP systems. Additionally similar to Spanglish, code-switching can occur at the morpheme level, where speakers often add morphological inflections to nouns.

Tasks
LinCE is built upon four tasks and four language pairs to provide a total of eleven datasets. In Sections 4.1 to 4.4, we discuss the datasets used for every task. Then, in Section 4.5, we describe and justify the modifications to nine out of the ten datasets in order to establish official splits that can be adopted for this benchmark. Lastly, in Section 4.6, we explain the evaluation criteria to rank the leaderboard in the LinCE platform.

Language Identification (LID)
Handling code-switched data requires to identify the languages involved. The task of language identification (LID) is one of the first steps that validates whether a system can handle code-switched data or not. Correctly classifying the language associated to text units (e.g., words or sub-word tokens) enables to process code-switched text in higher-level applications where general language understanding takes place. LinCE uses preexisting datasets for the language identification task. Specifically, in this version of LinCE, we focus on the language pairs Spanish-English, Hindi-English, Nepali-English, and Modern Standard Arabic-Egyptian Arabic. We briefly explain each corpus below, and for some of them, we propose new splits as explained in the stratification section (Section 4.5). Figure 1 shows the final distribution of the labels across the LID corpora used in LinCE. Also, these datasets follow the CALCS LID label scheme, which is lang1, lang2, mixed (partially in both languages), ambiguous (either one or the other language), fw (a language different than lang1 and lang2), ne (named entities), other, and unk (unrecognizable words). More details about the LID label scheme are in Appendix A.1.
• SPA-ENG. We use the Spanish-English corpus from the 2016 CALCS workshop (Molina et al., 2016). This corpus uses Twitter data and it contains 32,651 posts that are comprised of 390,953 tokens. We provide new splits for this corpus because the original splits do not have a similar label distribution and the label fw does not appear in the development set.
• HIN-ENG. We use the Hindi-English corpus released by Mave et al. (2018). This corpus uses Twitter and Facebook data, which have been partly collected and partly re-used from the ICON 2016 competition (Sequiera et al., 2015a). The corpus contains a total 7,421 posts comprised of 146,722 tokens. Also, we proceed with the stratification process on this corpus because the length of the posts were not considered while doing the splits; Twitter has a character length limit in its post, whereas Facebook posts do not have such restriction resulting in significantly longer text. Moreover, the labels ambiguous and unk do not appear in the development set.
• NEP-ENG. The Nepali-English corpus comes from the 2014 CALCS workshop (Solorio et al., 2014). This corpus was collected from Twitter and it contains 13,011 posts and 188,784 tokens. We perform a stratification process to provide standard splits for this corpus since the organizers only provided train and test, and the test set does not include any occurrence of the ambiguous class.
• MSA-EA. We use the Modern Standard Arabic-Egyptian Arabic corpus from the 2016 CALCS workshop (Molina et al., 2016). This corpus contains Twitter data and it is comprised of 11,243 tweets with 227,354 tokens. Note that there is no occurrence of the labels fw and unk in the entire corpus. We propose new partitions due to the variation across distributions for both the LID labels as well as sentence lengths.

Parts-of-Speech (POS) Tagging
Part-of-speech (POS) tagging is an important linguistic component that enables more sophisticated syntactic anal- ysis such as constituency and dependency parsing. Codeswitched data is not exempted of such analysis. In fact, previous studies have shown that syntax is preserved and compliant with the syntactic rules of the individual languages when code-switching occurs (Solorio and Liu, 2008). In this benchmark, we consider the language pairs Hindi-English and Spanish-English: • HIN-ENG. Singh et al. (2018b) provides 1,489 tweets (33,010 tokens) annotated with POS tags and three language IDs (hi for Hindi, en for English, and rest for any other token). The POS tags are annotated using the universal POS tagset proposed by Petrov et al. (2012) with the addition of two labels: PART NEG and PRON WH. The corpus does not provide training, development, and test splits due to the small number of samples. However, for the purposes of the benchmark, we propose standard splits using the stratification criteria discussed in Section 4.5.
• SPA-ENG. We use the Miami Bangor corpus with the annotations provided by Soto and Hirschberg (2017). The Bangor corpus is composed of bilingual conversations from four speakers with a total of 42,911 utterances and 333,069 tokens. The corpus contains POS tags from the universal POS tagset and LID labels. The LID labels are eng for English, spa for Spanish, eng&spa for mixed or ambiguous words, and UNK for everything else. Additionally, we proceed with the stratification process to provide the official training, development, and testing sets for this benchmark since the original sets were split by speakers.

Named Entity Recognition (NER)
Named entity recognition (NER) is another important core NLP task that enables higher-level applications such as question-answering, semantic role labeling, and information extraction. LinCE covers NER for three languages pairs: Spanish-English, Modern Standard Arabic-Egyptian Arabic, and Hindi-English. • SPA-ENG. This corpus was introduced in the 2018 CALCS competition for NER (Aguilar et al., 2018), and it contains a total of 67,223 tweets with 808,663 tokens. The labels are organization, person, location, group, product, title, event, time, and other. Along with the NER labels, we have added the LID categories for every token, which follows the CALCS LID scheme. Moreover, we propose new splits for this corpus since the distribution of the NER labels across the splits is not consistent to the one from the full corpus. Additionally, the original development set is significantly small compared to the other splits, only accounting for 832 tweets, and the LID labels were not taken into consideration for the splitting process (e.g., the label fw does not appear in the development set). We provide new splits following the stratified process described in Section 4.5 • MSA-EA. This corpus was also introduced in the 2018 CALCS competition for NER, following the same entity label scheme as in the SPA-ENG corpus. The corpus uses the tweets from the 2016 CALCS LID dataset to form the training and development sets. While the LID labels are available for the training and development splits, the test set was annotated only using the NER labels. Thus, this is the only corpus for which we do not consider the language identification analysis. The corpus contains 12,335 posts and 248,452 tokens. We adopt the splits provided by the organizers during the 2018 CALCS competition.
• HIN-ENG. This corpus is proposed by Singh et al. (2018a), and it is composed of 2,079 tweets with 35,374 tokens. The dataset has been annotated with both NER and LID labels. The entity labels are person, location, and organization, while the LID labels are eng (English), hin (Hindi), and rest (any other token). This dataset is small, and for that reason, the authors opted to do 5-fold cross validation instead of partitioning the dataset. Nevertheless, Figure 4: Label distribution of the sentiment analysis corpus used in LinCE. Note that this distribution differs from the origial dataset.
for the sake of the benchmark, we split the data using our stratification process that fairly splits the dataset accounting for LID and NER label distributions, as well as the distribution of the tweet lengths.

Sentiment Analysis (SA)
We choose sentiment analysis as our fourth benchmark task to incorporate a high-level NLP application in contrast to the previous core NLP tasks. We use the Spanish-English corpus provided in the SentiMix competition (Patwa et al., 2020). The organizers reduce the monolingual posts, increasing the number code-switched instances. Table 2 shows that this language pair is the second highest scores on the "All CMI" column for the sentiment analysis task. The task requires to predict one of the sentiments positive, negative, or neutral for every post. Additionally, this corpus is annotated with LID labels at the token level, following the CALCS LID scheme, and it contains 18,789 tweets comprised of 286,810 tokens. We propose new partitions for this dataset to correct the label distribution from the original splits.

Stratification
For nine out of ten datasets, 3 we propose new splits that in our view lead to a more appropriate evaluation (see Table 3 for a high-level distribution). We provide new splits for datasets where we found at least one of the following issues: 1. At least one of the splits does not have one or more classes. That is, one or more classes from are not evaluated at all in the development or test set.
2. The distribution of the label set for a given task is substantially different across splits or against the full corpus distribution (i.e., when merging all the splits into a single set).
3. The length of the sentences do not follow a similar distribution across splits or against the full corpus. This is a relevant criteria to consider since length is positively correlated with context, and less or more context can make a huge different for tasks such as NER.   The column Reason provides the reason number according to the aspects listed in Section 4.5. For the KLdivergence columns, we provide the average scores for the original (before) and the proposed (after) splits. The lower the score, the more similar the splits are to the full corpus distribution.
4. The NER, POS, and SA datasets contain LID labels that were not considered during the time of the stratification process, potentially affecting the balance of code-switching occurrences across the splits.
5. There is no official split for the training, development and test sets to provide the scope of fair comparison.
For the datasets where we find at least one of these issues (see Table 4), we proceed to stratify based on the language identification labels, if available, the task-specific labels (i.e., for tasks other than LID), and the lengths of the sentences. Note that providing splits that consider these three factors jointly in the stratification process is not trivial. In fact, in the case of sequence labeling tasks, we may have multiple non-unique labels per sentence, which constraints the ability to draw a distribution similar to the full corpora (e.g., adding a sentence impacts the distribution of different labels occurring in the same sentence).
To provide splits considering these three criteria, we follow the iterative stratification process proposed by Sechidis et al. (2011). This process targets multi-label data, which is a different scenario for the document and sequence labeling classification datasets used in LinCE. To adapt the sequence labels to the multi-label scenario, we treat a post as a single sample that is associated to a group of labels. In the case of tasks other than LID, we gather the LID labels with the task-specific labels (i.e., NER, POS, or SA labels) into a single group of unique labels. We also incorporate sentence lengths to the label set of a sample by choosing one of three length categories: small (≤ 10 tokens), medium (>10 and ≤ 20 tokens), or large (>20 tokens). For instance, the SPA-ENG NER sample "LREC event ne será lang2 hosted in Marseille location ne " English: "LREC will be hosted in Marseille" has the set of unique labels {lang1, lang2, ne, event, location, O, small}, where the first three labels are for LID, the following three labels are for NER, and the last one represents the sentence length (note that the order and the repetitions of the labels do not matter). Once we have the set of labels associated to a single sample (e.g., a set of LID, POS, and length labels), we can follow the iterative stratification processed used for multi-label classification on our corpus. We have found that this procedure works well in practice; we measure the KL-divergence of the label distributions from each of the splits against the distribution of the full corpus before and after the stratification, and we found that the proposed splits have less divergence (see Table 4). While KL-divergence is not often employed to corroborate the distributions of a stratified corpus, we use the divergence score to quantify whether the distribution of the full corpus has been preserved in the proposed splits, and whether the new splits are better distributions than the original splits. The final numbers of sentences and tokens per partition are listed in Table 3.

Evaluation
LinCE adopts an evaluation model similar to SemEval, Kaggle, and GLUE (Wang et al., 2018a). A platform will be hosted at ritual.uh.edu/lince where participants will be able to upload their predictions for the test data on each task. The platform will score the submissions and publish the results in a public leaderboard for each task. The leaderboard is ranked by the average of the task scores.

Baseline Models
We propose LinCE to motivate a single model architecture that has good generalization capability across all the proposed tasks. To this end, we experiment with the following model architectures that demonstrated superior performance in a wide range of NLP tasks in recent years.
• Bidirectional LSTM. Our simplest baseline is based on LSTM cells (Hochreiter and Schmidhuber, 1997), which we operate towards both directions of the texts. For the sequence labeling tasks (i.e., LID, POS, and NER), we concatenate the forward and backward hidden representations for each of the input token and use a linear layer to predict the most probable label for each token. For sentiment analysis, we perform a maxpooling operation over all the hidden representations of the tokens and use that to predict the most probable sentiment class. For all of the tasks, we represent each token by a randomly initialized word embedding vector that is tuned during the training process.
• ELMo. Combining character-level convolution and word-level sequence modeling with LSTM, ELMo  have shown improvement in various NLP tasks acting as a pre-trained language model. We fine-tune the publicly available pre-trained ELMo models on the proposed tasks by using its high-level word representations to perform sequence labeling (i.e., LID, POS, and NER). We use the mean of the token representations to predict sentiment labels for the SA task.
• ML-BERT Transformer (Vaswani et al., 2017) based pre-trained language model like BERT (Devlin et al., 2018) have shown impressive generalized performance in a wide range of natural language understanding tasks. The strength of such model comes from the large amount of parameters tuned on a huge amount of training data from diverse domains. We develop our third baseline system with the pre-trained BERT model trained on multilingual data from 104 languages. We add a task-specific prediction layer over BERT and fine-tune the whole model.

Implementation and Training
We implement the models using PyTorch 4 deep learning library. We also use AllenNLP  and HuggingFace's Transformers (Wolf et al., 2019) to finetune pre-trained ELMo and BERT based models, respectively. While evaluating a single model architecture across 4 pytorch.org different tasks, we keep the whole model architecture exactly the same except the prediction layer. We train the BiLSTM and ELMo models using SGD (η = 0.1 for BiL-STM and η = 0.01 for ELMo, β = 0.9). For fine-tuning BERT, we use the AdamW (Loshchilov and Hutter, 2019) (η = 5e −5 , = 1e −8 ). We use a batch size of 32 for training every model. We train each model for a maximum of 50 epochs and stop if the validation performance does not improve for 10 epochs. All of our experimental choices were tuned by observing the performance on the validation sets.

Results and Analysis
We report our results on the test sets for the LinCE tasks in Table 5 and discuss the details here. Across almost all the tasks we observe superior performance of the pretrained language models compared to the simple BiLSTM model. Among the pre-trained language models, ML-BERT demonstrates superior performance in each task for all the available language pairs. ELMo's performance is very close to ML-BERT in most of the LID and POS tasks (≈1-4%), but the performance gap is bigger for NER (≈8-10%) and SA (≈18%). The average performance gap between ELMo and BERT is ≈6%. We suspect that such improvement for ML-BERT against ELMo is powered by its larger parameter set (110M vs 13.6M) and the amount of data used for pre-training (Book Corpus (Zhu et al., 2015) and Wikipedia for BERT and Billion Words Benchmark (Chelba et al., 2013) for ELMo). It is also noteworthy to mention that the training data for BERT is document-level data, whereas the Billion Words Benchmark is a sentencelevel corpus. Among the four tasks, NER and SA seem harder compared to LID and POS. It shows that the involvement of semantic understanding in code-switched texts makes tasks harder compared to syntactic analysis. We observe that the LID task is harder for MSA-EA compared to the other language pairs. A possible reason for this is the large overlap between these two languages, which also affected the annotation process (Molina et al., 2016).

Conclusion
In this paper, we motivate the need for a centralized platform to perform evaluation of technology for codeswitching data across multiple tasks and language pairs. To this end, we introduce the Linguistic Code-switching Evaluation (LinCE) benchmark using ten publicly available datasets. In addition, we review such datasets and found important issues that undermine the evaluation process (e.g., labels not appearing in the test set, or substantially different distributions among splits, etc.). Then, we propose new splits using a new stratification technique with up to three criteria (e.g., LID labels, task-specific labels, and sentence lengths). We show the distribution of the full corpus is preserved in the proposed splits used in LinCE, which is not always the case in the original partitions. Additionally, we provide results with strong baselines using state-of-the-art models on monolingual datasets, including BERT and ELMo. Finally, we expect that LinCE will be well-received by the NLP community, and we will keep the platform evolving with the incorporation of more tasks and language pairs in the near future.

Acknowledgements
This work was supported by the National Science Foundation (NSF) on the grant #1910192. We thank the authors of the individual datasets that agreed to centralize their corpora into a single benchmark for code-switching. Also, we thank the undergraduate students at the University of Houston, Jason Ho, Tarun Appannagari, and Dwija Parikh, that helped on the creation of the LinCE benchmark website.