Evaluating Gender Bias in Hindi-English Machine Translation
  With language models being deployed increasingly in the real world, it is
essential to address the issue of the fairness of their outputs. The word
embedding representations of these language models often implicitly draw
unwanted associations that form a social bias within the model. The nature of
gendered languages like Hindi, poses an additional problem to the
quantification and mitigation of bias, owing to the change in the form of the
words in the sentence, based on the gender of the subject. Additionally, there
is sparse work done in the realm of measuring and debiasing systems for Indic
languages. In our work, we attempt to evaluate and quantify the gender bias
within a Hindi-English machine translation system. We implement a modified
version of the existing TGBI metric based on the grammatical considerations for
Hindi. We also compare and contrast the resulting bias measurements across
multiple metrics for pre-trained embeddings and the ones learned by our machine
translation model.
