UserBERT: Modeling Long- and Short-Term User Preferences via
  Self-Supervision
  E-commerce platforms generate vast amounts of customer behavior data, such as
clicks and purchases, from millions of unique users every day. However,
effectively using this data for behavior understanding tasks is challenging
because there are usually not enough labels to learn from all users in a
supervised manner. This paper extends the BERT model to e-commerce user data
for pre-training representations in a self-supervised manner. By viewing user
actions in sequences as analogous to words in sentences, we extend the existing
BERT model to user behavior data. Further, our model adopts a unified structure
to simultaneously learn from long-term and short-term user behavior, as well as
user attributes. We propose methods for the tokenization of different types of
user behavior sequences, the generation of input representation vectors, and a
novel pretext task to enable the pre-trained model to learn from its own input,
eliminating the need for labeled training data. Extensive experiments
demonstrate that the learned representations result in significant improvements
when transferred to three different real-world tasks, particularly compared to
task-specific modeling and multi-task representation learning
