How to Measure Gender Bias in Machine Translation: Optimal Translators,
  Multiple Reference Points
  In this paper, as a case study, we present a systematic study of gender bias
in machine translation with Google Translate. We translated sentences
containing names of occupations from Hungarian, a language with gender-neutral
pronouns, into English. Our aim was to present a fair measure for bias by
comparing the translations to an optimal non-biased translator. When assessing
bias, we used the following reference points: (1) the distribution of men and
women among occupations in both the source and the target language countries,
as well as (2) the results of a Hungarian survey that examined if certain jobs
are generally perceived as feminine or masculine. We also studied how expanding
sentences with adjectives referring to occupations effect the gender of the
translated pronouns. As a result, we found bias against both genders, but
biased results against women are much more frequent. Translations are closer to
our perception of occupations than to objective occupational statistics.
Finally, occupations have a greater effect on translation than adjectives.
