Online Evaluation Methods for the Causal Effect of Recommendations
  Evaluating the causal effect of recommendations is an important objective
because the causal effect on user interactions can directly leads to an
increase in sales and user engagement. To select an optimal recommendation
model, it is common to conduct A/B testing to compare model performance.
However, A/B testing of causal effects requires a large number of users, making
such experiments costly and risky. We therefore propose the first interleaving
methods that can efficiently compare recommendation models in terms of causal
effects. In contrast to conventional interleaving methods, we measure the
outcomes of both items on an interleaved list and items not on the interleaved
list, since the causal effect is the difference between outcomes with and
without recommendations. To ensure that the evaluations are unbiased, we either
select items with equal probability or weight the outcomes using inverse
propensity scores. We then verify the unbiasedness and efficiency of online
evaluation methods through simulated online experiments. The results indicate
that our proposed methods are unbiased and that they have superior efficiency
to A/B testing.
