Privileged Features Distillation at Taobao Recommendations
  Features play an important role in the prediction tasks of e-commerce
recommendations. To guarantee the consistency of off-line training and on-line
serving, we usually utilize the same features that are both available. However,
the consistency in turn neglects some discriminative features. For example,
when estimating the conversion rate (CVR), i.e., the probability that a user
would purchase the item if she clicked it, features like dwell time on the item
detailed page are informative. However, CVR prediction should be conducted for
on-line ranking before the click happens. Thus we cannot get such post-event
features during serving.
  We define the features that are discriminative but only available during
training as the privileged features. Inspired by the distillation techniques
which bridge the gap between training and inference, in this work, we propose
privileged features distillation (PFD). We train two models, i.e., a student
model that is the same as the original one and a teacher model that
additionally utilizes the privileged features. Knowledge distilled from the
more accurate teacher is transferred to the student to improve its accuracy.
During serving, only the student part is extracted and it relies on no
privileged features. We conduct experiments on two fundamental prediction tasks
at Taobao recommendations, i.e., click-through rate (CTR) at coarse-grained
ranking and CVR at fine-grained ranking. By distilling the interacted features
that are prohibited during serving for CTR and the post-event features for CVR,
we achieve significant improvements over their strong baselines. During the
on-line A/B tests, the click metric is improved by +5.0% in the CTR task. And
the conversion metric is improved by +2.3% in the CVR task. Besides, by
addressing several issues of training PFD, we obtain comparable training speed
as the baselines without any distillation.
