Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual
  Representations
  Large-scale pretraining of visual representations has led to state-of-the-art
performance on a range of benchmark computer vision tasks, yet the benefits of
these techniques at extreme scale in complex production systems has been
relatively unexplored. We consider the case of a popular visual discovery
product, where these representations are trained with multi-task learning, from
use-case specific visual understanding (e.g. skin tone classification) to
general representation learning for all visual content (e.g. embeddings for
retrieval). In this work, we describe how we (1) generate a dataset with over a
billion images via large weakly-supervised pretraining to improve the
performance of these visual representations, and (2) leverage Transformers to
replace the traditional convolutional backbone, with insights into both system
and performance improvements, especially at 1B+ image scale. To support this
backbone model, we detail a systematic approach to deriving weakly-supervised
image annotations from heterogenous text signals, demonstrating the benefits of
clustering techniques to handle the long-tail distribution of image labels.
Through a comprehensive study of offline and online evaluation, we show that
large-scale Transformer-based pretraining provides significant benefits to
industry computer vision applications. The model is deployed in a production
visual shopping system, with 36% improvement in top-1 relevance and 23%
improvement in click-through volume. We conduct extensive experiments to better
understand the empirical relationships between Transformer-based architectures,
dataset scale, and the performance of production vision systems.
