User Acceptance of Gender Stereotypes in Automated Career
  Recommendations
  Currently, there is a surge of interest in fair Artificial Intelligence (AI)
and Machine Learning (ML) research which aims to mitigate discriminatory bias
in AI algorithms, e.g. along lines of gender, age, and race. While most
research in this domain focuses on developing fair AI algorithms, in this work,
we show that a fair AI algorithm on its own may be insufficient to achieve its
intended results in the real world. Using career recommendation as a case
study, we build a fair AI career recommender by employing gender debiasing
machine learning techniques. Our offline evaluation showed that the debiased
recommender makes fairer career recommendations without sacrificing its
accuracy. Nevertheless, an online user study of more than 200 college students
revealed that participants on average prefer the original biased system over
the debiased system. Specifically, we found that perceived gender disparity is
a determining factor for the acceptance of a recommendation. In other words,
our results demonstrate we cannot fully address the gender bias issue in AI
recommendations without addressing the gender bias in humans.
