Detect Language of Transliterated Texts
  Informal transliteration from other languages to English is prevalent in
social media threads, instant messaging, and discussion forums. Without
identifying the language of such transliterated text, users who do not speak
that language cannot understand its content using translation tools. We propose
a Language Identification (LID) system, with an approach for feature
extraction, which can detect the language of transliterated texts reasonably
well even with limited training data and computational resources. We tokenize
the words into phonetic syllables and use a simple Long Short-term Memory
(LSTM) network architecture to detect the language of transliterated texts.
With intensive experiments, we show that the tokenization of transliterated
words as phonetic syllables effectively represents their causal sound patterns.
Phonetic syllable tokenization, therefore, makes it easier for even simpler
model architectures to learn the characteristic patterns to identify any
language.
