Indic-Transformers: An Analysis of Transformer Language Models for
  Indian Languages
  Language models based on the Transformer architecture have achieved
state-of-the-art performance on a wide range of NLP tasks such as text
classification, question-answering, and token classification. However, this
performance is usually tested and reported on high-resource languages, like
English, French, Spanish, and German. Indian languages, on the other hand, are
underrepresented in such benchmarks. Despite some Indian languages being
included in training multilingual Transformer models, they have not been the
primary focus of such work. In order to evaluate the performance on Indian
languages specifically, we analyze these language models through extensive
experiments on multiple downstream tasks in Hindi, Bengali, and Telugu
language. Here, we compare the efficacy of fine-tuning model parameters of
pre-trained models against that of training a language model from scratch.
Moreover, we empirically argue against the strict dependency between the
dataset size and model performance, but rather encourage task-specific model
and method selection. We achieve state-of-the-art performance on Hindi and
Bengali languages for text classification task. Finally, we present effective
strategies for handling the modeling of Indian languages and we release our
model checkpoints for the community :
https://huggingface.co/neuralspace-reverie.
