Measuring Recommender System Effects with Simulated Users
  Imagine a food recommender system -- how would we check if it is
\emph{causing} and fostering unhealthy eating habits or merely reflecting
users' interests? How much of a user's experience over time with a recommender
is caused by the recommender system's choices and biases, and how much is based
on the user's preferences and biases? Popularity bias and filter bubbles are
two of the most well-studied recommender system biases, but most of the prior
research has focused on understanding the system behavior in a single
recommendation step. How do these biases interplay with user behavior, and what
types of user experiences are created from repeated interactions?
  In this work, we offer a simulation framework for measuring the impact of a
recommender system under different types of user behavior. Using this
simulation framework, we can (a) isolate the effect of the recommender system
from the user preferences, and (b) examine how the system performs not just on
average for an "average user" but also the extreme experiences under atypical
user behavior. As part of the simulation framework, we propose a set of
evaluation metrics over the simulations to understand the recommender system's
behavior. Finally, we present two empirical case studies -- one on traditional
collaborative filtering in MovieLens and one on a large-scale production
recommender system -- to understand how popularity bias manifests over time.
