Revisiting Popularity and Demographic Biases in Recommender Evaluation
  and Effectiveness
  Recommendation algorithms are susceptible to popularity bias: a tendency to
recommend popular items even when they fail to meet user needs. A related issue
is that the recommendation quality can vary by demographic groups. Marginalized
groups or groups that are under-represented in the training data may receive
less relevant recommendations from these algorithms compared to others. In a
recent study, Ekstrand et al. investigate how recommender performance varies
according to popularity and demographics, and find statistically significant
differences in recommendation utility between binary genders on two datasets,
and significant effects based on age on one dataset. Here we reproduce those
results and extend them with additional analyses. We find statistically
significant differences in recommender performance by both age and gender. We
observe that recommendation utility steadily degrades for older users, and is
lower for women than men. We also find that the utility is higher for users
from countries with more representation in the dataset. In addition, we find
that total usage and the popularity of consumed content are strong predictors
of recommender performance and also vary significantly across demographic
groups.
