Evaluating Input Representation for Language Identification in
  Hindi-English Code Mixed Text
  Natural language processing (NLP) techniques have become mainstream in the
recent decade. Most of these advances are attributed to the processing of a
single language. More recently, with the extensive growth of social media
platforms focus has shifted to code-mixed text. The code-mixed text comprises
text written in more than one language. People naturally tend to combine local
language with global languages like English. To process such texts, current NLP
techniques are not sufficient. As a first step, the text is processed to
identify the language of the words in the text. In this work, we focus on
language identification in code-mixed sentences for Hindi-English mixed text.
The task of language identification is formulated as a token classification
task. In the supervised setting, each word in the sentence has an associated
language label. We evaluate different deep learning models and input
representation combinations for this task. Mainly, character, sub-word, and
word embeddings are considered in combination with CNN and LSTM based models.
We show that sub-word representation along with the LSTM model gives the best
results. In general sub-word representations perform significantly better than
other input representations. We report the best accuracy of 94.52% using a
single layer LSTM model on the standard SAIL ICON 2017 test set.
