{'fraction_non_alphanumeric': 0.0369549342791571, 'fraction_numerical': 0.026549134153974547, 'mean_word_length': 4.791421235462921, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Big Transfer (BiT): General Visual Representation Learning

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.


Introduction
Strong performance using deep learning usually requires a large amount of taskspecific data and compute. These per-task requirements can make new tasks prohibitively expensive. Transfer learning offers a solution: task-specific data and compute are replaced with a pre-training phase. A network is trained once on a large, generic dataset, and its weights are then used to initialize subsequent tasks which can be solved with fewer data points, and less compute [40,44,14].
We revisit a simple paradigm: pre-train on a large supervised source dataset, and fine-tune the weights on the target task. Numerous improvements to deep network training have recently been introduced, e.g. [55,62,26,35,22,1,64,67,54,60]. We aim not to introduce a new component or complexity, but to provide a recipe that uses the minimal number of tricks yet attains excellent performance on many tasks. We call this recipe "Big Transfer" (BiT).
We train networks on three different scales of datasets. The largest, BiT-L is trained on the JFT-300M dataset [51], which contains 300 M noisily labelled images. We transfer BiT to many diverse tasks; with training set sizes ranging from 1 example per class to 1M total examples. These tasks include Im-ageNet's ILSVRC-2012 [10], CIFAR-10/100 [27], Oxford-IIIT Pet [41], Oxford Flowers-102 [39] (including few-shot variants), and the 1000-sample VTAB-1k benchmark [66], which consists of 19 diverse datasets. BiT-L attains state-ofthe-art performance on many of these tasks, and is surprisingly effective when very little downstream data is available ( Figure 1). We also train BiT-M on the public ImageNet-21k dataset, and attain marked improvements over the popular ILSVRC-2012 pre-training. Importantly, BiT only needs to be pre-trained once and subsequent finetuning to downstream tasks is cheap. By contrast, other state-of-the-art methods require extensive training on support data conditioned on the task at hand [38, 61,63]. Not only does BiT require a short fine-tuning protocol for each new task, but BiT also does not require extensive hyperparameter tuning on new tasks. Instead, we present a heuristic for setting the hyperparameters for transfer, which works well on our diverse evaluation suite.
We highlight the most important components that make Big Transfer effective, and provide insight into the interplay between scale, architecture, and training hyperparameters. For practitioners, we will release the performant BiT-M model trained on ImageNet-21k.

Big Transfer
We review the components that we found necessary to build an effective network for transfer. Upstream components are those used during pre-training, and downstream are those used during fine-tuning to a new task.

Upstream Pre-Training
The first component is scale. It is well-known in deep learning that larger networks perform better on their respective tasks [15,48]. Further, it is recognized that larger datasets require larger architectures to realize benefits, and vice versa [25,45]. We study the effectiveness of scale (during pre-training) in the context of transfer learning, including transfer to tasks with very few datapoints. We investigate the interplay between computational budget (training time), architecture size, and dataset size. For this, we train three BiT models on three large datasets: ILSVRC-2012 [46] which contains 1.3M images (BiT-S), ImageNet-21k [10] which contains 14M images (BiT-M), and JFT [51] which contains 300M images (BiT-L).
The second component is Group Normalization (GN) [60] and Weight Standardization (WS) [34]. Batch Normalization (BN) [21] is used in most state-ofthe-art vision models to stabilize training. However, we find that BN is detrimental to Big Transfer for two reasons. First, when training large models with small per-device batches, BN performs poorly or incurs inter-device synchronization cost. Second, due to the requirement to update running statistics, BN is detrimental for transfer. GN, when combined with WS, has been shown to improve performance on small-batch training for ImageNet and COCO [34]. Here, we show that the combination of GN and WS is useful for training with large batch sizes, and has a significant impact on transfer learning.

Transfer to Downstream Tasks
We propose a cheap fine-tuning protocol that applies to many diverse downstream tasks. Importantly, we avoid expensive hyperparameter search for every new task and dataset size; we try only one hyperparameter per task. We use a heuristic rule-which we call BiT-HyperRule-to select the most important hyperparameters for tuning as a simple function of the task's intrinsic image resolution and number of datapoints. We found it important to set the following hyperparameters per-task: training schedule length, resolution, and whether to use MixUp regularization [67]. We use BiT-HyperRule for over 20 tasks in this paper, with training sets ranging from 1 example per class to over 1M total examples. The exact settings for BiT-HyperRule are presented in Section 3.3.
During fine-tuning, we use the following standard data pre-processing: we resize the image to a square, crop out a smaller random square, and randomly horizontally flip the image at training time. At test time, we only resize the image to a fixed size. In some tasks horizontal flipping or cropping destroys the label semantics, making the task impossible. An example is if the label requires predicting object orientation or coordinates in pixel space. In these cases we omit flipping or cropping when appropriate.
Recent work has shown that existing augmentation methods introduce inconsistency between training and test resolutions for CNNs [57]. Therefore, it is common to scale up the resolution by a small factor at test time. As an alternative, one can add a step at which the trained model is fine-tuned to the test resolution [57]. The latter is well-suited for transfer learning; we include the resolution change during our fine-tuning step.
We found that MixUp [67] is not useful for pre-training BiT, likely due to the abundance of data. However, it is sometimes useful for transfer. Interestingly, it is most useful for mid-sized datasets, and not for few-shot transfer, see Section 3.3 for where we apply MixUp.
Surprisingly, we do not use any of the following forms of regularization during downstream tuning: weight decay to zero, weight decay to initial parameters [31], or dropout. Despite the fact that the network is very large-BiT has 928 million parameters-the performance is surprisingly good without these techniques and their respective hyperparameters, even when transferring to very small datasets. We find that setting an appropriate schedule length, i.e. training longer for larger datasets, provides sufficient regularization.

Experiments
We train three upstream models using three datasets at different scales: BiT-S, BiT-M, and BiT-L. We evaluate these models on many downstream tasks and attain very strong performance on high and low data regimes.

Data for Upstream Training
BiT-S is trained on the ILSVRC-2012 variant of ImageNet, which contains 1.28 million images and 1000 classes. Each image has a single label. BiT-M is trained on the full ImageNet-21k dataset [10], a public dataset containing 14.2 million images and 21k classes organized by the WordNet hierarchy. Images may contain multiple labels. BiT-L is trained on the JFT-300M dataset [51,38,61]. This dataset is a newer version of that used in [18,8]. JFT-300M consists of around 300 million images with 1.26 labels per image on average. The labels are organized into a hierarchy of 18 291 classes. Annotation is performed using an automatic pipeline, and are therefore imperfect; approximately 20% of the labels are noisy. We remove all images present in downstream test sets from JFT-300M. See Supplementary Material section C for details. Note: the "-S/M/L" suffix refers to the pre-training datasets size and schedule, not architecture. We train BiT with several architecture sizes, the default (largest) being ResNet152x4.

Downstream Tasks
We evaluate BiT on long-standing benchmarks: ILSVRC-2012 [ . These datasets differ in the total number of images, input resolution and nature of their categories, from general object categories in ImageNet and CIFAR to fine-grained ones in Pets and Flowers. We fine-tune BiT on the official training split and report results on the official test split if publicly available. Otherwise, we use the val split. Table 1: Top-1 accuracy for BiT-L on many datasets using a single model and single hyperparameter setting per task (BiT-HyperRule). The entries show median ± standard deviation across 3 fine-tuning runs. Specialist models are those that condition pre-training on each task, while generalist models, including BiT, perform task-independent pre-training. ( Concurrent work.) BiT To further assess the generality of representations learned by BiT, we evaluate on the Visual Task Adaptation Benchmark (VTAB) [66]. VTAB consists of 19 diverse visual tasks, each of which has 1000 training samples (VTAB-1k variant). The tasks are organized into three groups: natural, specialized and structured. The VTAB-1k score is top-1 recognition performance averaged over these 19 tasks. The natural group of tasks contains classical datasets of natural images captured using standard cameras. The specialized group also contains images captured in the real world, but through specialist equipment, such as satellite or medical images. Finally, the structured tasks assess understanding of the the structure of a scene, and are mostly generated from synthetic environments. Example structured tasks include object counting and 3D depth estimation.

Hyperparameter Details
Upstream Pre-Training All of our BiT models use a vanilla ResNet-v2 architecture [16], except that we replace all Batch Normalization [21] layers with Group Normalization [60] and use Weight Standardization [43] in all convolutional layers. See Section 4.3 for analysis. We train ResNet-152 architectures in all datasets, with every hidden layer widened by a factor of four (ResNet152x4). We study different model sizes and the coupling with dataset size in Section 4.1.
We train all of our models upstream using SGD with momentum. We use an initial learning rate of 0.03, and momentum 0.9. During image preprocessing stage we use image cropping technique from [53] and random horizontal mirroring followed by 224 × 224 image resize. We train both BiT-S and BiT-M for 90 epochs and decay the learning rate by a factor of 10 at 30, 60 and 80 epochs. For BiT-L, we train for 40 epochs and decay the learning rate after 10, 23, 30 and 37 epochs. We use a global batch size of 4096 and train on a Cloud TPUv3-512 [24], resulting in 8 images per chip. We use linear learning rate warm-up for 5000 optimization steps and multiply the learning rate by batch size 256 following [11]. During pre-training we use a weight decay of 0.0001, but as discussed in Section 2, we do not use any weight decay during transfer. For all tasks, we use SGD with an initial learning rate of 0.003, momentum 0.9, and batch size 512. We resize input images with area smaller than 96 × 96 pixels to 160 × 160 pixels, and then take a random crop of 128 × 128 pixels. We resize larger images to 448 × 448 and take a 384 × 384-sized crop. 1 We apply random crops and horizontal flips for all tasks, except those for which cropping or flipping destroys the label semantics, see Supplementary section F for details.
For schedule length, we define three scale regimes based on the number of examples: we call small tasks those with fewer than 20 k labeled examples, medium those with fewer than 500 k, and any larger dataset is a large task. We fine-tune BiT for 500 steps on small tasks, for 10k steps on medium tasks, and for 20k steps on large tasks. During fine-tuning, we decay the learning rate by a factor of 10 at 30%, 60% and 90% of the training steps. Finally, we use MixUp [67], with α = 0.1, for medium and large tasks. See Supplementary section A for details.

Standard Computer Vision Benchmarks
We evaluate BiT-L on standard benchmarks and compare its performance to the current state-of-the-art results (Table 1). We separate models that perform taskindependent pre-training ("general" representations), from those that perform task-dependent auxiliary training ("specialist" representations). The specialist methods condition on a particular task, for example ILSVRC-2012, then train using a large support dataset, such as JFT-300M [38] or Instagram-1B [63]. See discussion in Section 5. Specialist representations are highly effective, but require a large training cost per task. By contrast, generalized representations require large-scale training only once, followed by a cheap adaptation phase.
BiT-L outperforms previously reported generalist SOTA models as well as, in many cases, the SOTA specialist models. Inspired by strong results of BiT-L trained on JFT-300M, we also train models on the public ImageNet-21k dataset. Each point represents the result after training on a balanced random subsample of the dataset (5 subsamples per dataset). The median across runs is highlighted by the curves. The variance across data samples is usually low, with the exception of 1-shot CIFAR-10, which contains only 10 images. Right: We summarize the state-of-the-art in semi-supervised learning as reference points. Note that a direct comparison is not meaningful; unlike BiT, semi-supervised methods have access to extra unlabelled data from the training distribution, but they do not make use of out-of-distribution labeled data.
This dataset is more than 10 times bigger than ILSVRC-2012, but it is mostly overlooked by the research community. In Table 2 we demonstrate that BiT-M trained on ImageNet-21k leads to substantially improved visual representations compared to the same model trained on ILSVRC-2012 (BiT-S), as measured by all our benchmarks. In Section 4.2, we discuss pitfalls that may have hindered wide adoption of ImageNet-21k as a dataset model for pre-training and highlight crucial components of BiT that enabled success on this large dataset. For completeness, we also report top-5 accuracy on ILSVRC-2012 with median ± standard deviation format across 3 runs: 98.46% ± 0.02% for BiT-L, 97.69% ± 0.02% for BiT-M and 95.65% ± 0.03% for BiT-S.

Tasks with Few Datapoints
We study the number of downstream labeled samples required to transfer BiT-L successfully. We transfer BiT-L using subsets of ILSVRC-2012, CIFAR-10, and CIFAR-100, down to 1 example per class. We also evaluate on a broader suite of 19 VTAB-1k tasks, each of which has 1000 training examples. Figure 2 (left half) shows BiT-L using few-shots on ILSVRC-2012, CIFAR-10, and CIFAR-100. We run multiple random subsamples, and plot every trial. Surprisingly, even with very few samples per class, BiT-L demonstrates strong performance and quickly approaches performance of the full-data regime. In particular, with just 5 labeled samples per class it achieves top-1 accuracy of 72.0% on ILSVRC-2012 and with 100 samples the top-1 accuracy goes to 84.1%. On CIFAR-100, we achieve 82.6% with just 10 samples per class. Semi-supervised learning also tackles learning with few labels. However, such approaches are not directly comparable to BiT. BiT uses extra labelled out-ofdomain data, whereas semi-supervised learning uses extra unlabelled in-domain data. Nevertheless, it is interesting to observe the relative benefits of transfer from out-of-domain labelled data versus in-domain semi-supervised data. In Figure 2 we show state-of-the-art results from the semi-supervised learning. Figure 3 shows the performance of BiT-L on the 19 VTAB-1k tasks. BiT-L with BiT-HyperRule substantially outperforms the previously reported state-ofthe-art. When looking into performance of VTAB-1k task subsets, BiT is the best on natural, specialized and structured tasks. The recently-proposed VIVI-Ex-100% [58] model that employs video data during upstream pre-training shows very similar performance on the structured tasks.
We investigate heavy per-task hyperparameter tuning in Supplementary Material section A and conclude that this further improves performance. We evaluate BiT on the new test-only ObjectNet dataset [2]. Importantly, this dataset closely resembles real-life scenarios, where object categories may appear in non-canonical context, viewpoint, rotation, etc. There are 313 object classes in total, with 113 overlapping with ILSVRC-2012. We follow the literature [2,6] and evaluate our models on those 113 classes. Figure 4 shows that larger architectures and pre-training on more data results in higher accuracies. Crucially, our results highlight that scaling both is essential for achieving unprecedented top-5 accuracy of 80.0%, an almost 25% absolute improvement over the previous state-of-the-art. We provide numeric results and additional results when classifying individual object bounding boxes [6] in the Supplementary Material section B.

Object Detection
Finally, we evaluate BiT on object detection. We use the COCO-2017 dataset [34] and train a top-performing object detector, RetinaNet [33], using pre-trained BiT models as backbones. Due to memory constraints, we use the ResNet-101x3 architecture for all of our BiT models. We fine-tune the detection models on the COCO-2017 train split and report results on the validation split using the standard metric [34] in Table 3  section E for details. Table 3 demonstrates that BiT models outperform standard ImageNet pretrained models. We can see clear benefits of pre-training on large data beyond ILSVRC-2012: pretraining on ImageNet-21k results in a 1.5 point improvement in Average Precision (AP), while pretraining on JFT-300M further improves performance by 0.6 points.

Analysis
We analyse various components of BiT: we demonstrate the importance of model capacity, discuss practical optimization caveats and choice of normalization layer.

Scaling Models and Datasets
The general consensus is that larger neural networks result in better performance. We investigate the interplay between model capacity and upstream dataset size on downstream performance. We evaluate the BiT models of different sizes (ResNet-50x1, ResNet-50x3, ResNet-101x1, ResNet-101x3, and ResNet-152x4) trained on ILSVRC-2012, ImageNet-21k, and JFT-300M on various downstream benchmarks. These results are summarized in Figure 5.  When pre-training on ILSVRC-2012, the benefit from larger models diminishes. However, the benefits of larger models are more pronounced on the larger two datasets. A similar effect is observed when training on Instagram hashtags [36] and in language modelling [25].
Not only is there limited benefit of training a large model size on a small dataset, but there is also limited (or even negative) benefit from training a small model on a larger dataset. Perhaps surprisingly, the ResNet-50x1 model trained on the JFT-300M dataset can even performs worse than the same architecture trained on the smaller ImageNet-21k. Thus, if one uses only a ResNet50x1, one may conclude that scaling up the dataset does not bring any additional benefits. However, with larger architectures, models pre-trained on JFT-300M significantly outperform those pre-trained on ILSVRC-2012 or ImageNet-21k. Figure 2 shows that BiT-L attains strong results even on tiny downstream datasets. Figure 6 ablates few-shot performance across different pre-training datasets and architectures. In the extreme case of one example per class, larger architectures outperform smaller ones when pre-trained on large upstream data. Interestingly, on ILSVRC-2012 with few shots, BiT-L trained on JFT-300M outperforms the models trained on the entire ILSVRC-2012 dataset itself. Note that for comparability, the classifier head is re-trained from scratch during fine-tuning, even when transferring ILSVRC-2012 full to ILSVRC-2012 few shot.

Optimization on Large Datasets
For standard computer vision datasets such as ILSVRC-2012, there are wellknown training procedures that are robust and lead to good performance. Progress in high-performance computing has made it feasible to learn from much larger datasets, such as ImageNet-21k, which has 14.2M images compared to ILSVRC-2012's 1.28M. However, there are no established procedures for training from such large datasets. In this section we provide some guidelines. Another important aspect of pre-training with large datasets is the weight decay. Lower weight decay can result in an apparent acceleration of convergence, Figure 7 rightmost plot. However, this setting eventually results in an under-performing final model. This counter-intuitive behavior stems from the interaction of weight decay and normalization layers [29,32]. Low weight decay results in growing weight norms, which in turn results in a diminishing effective learning rate. Initially this effect creates an impression of faster convergence, but it eventually prevents further progress. A sufficiently large weight decay is required to avoid this effect, and throughout we use 10 −4 .
Finally, we note that in all of our experiments we use stochastic gradient descent with momentum without any modifications. In our preliminary experiments we did not observe benefits from more involved adaptive gradient methods.

Large Batches, Group Normalization, Weight Standardization
Currently, training on large datasets is only feasible using many hardware accelerators. Data parallelism is the most popular distribution strategy, and this naturally entails large batch sizes. Many known algorithms for training with large batch sizes use Batch Normalization (BN) [21] as a component [12] or even highlight it as the key instrument required for large batch training [9]. Our larger models have a high memory requirement for any single accelerator chip, which necessitates small per-device batch sizes. However, BN performs worse when the number of images on each accelerator is too low [20]. An alternative strategy is to accumulate BN statistics across all of the accelerators. However, this has two major drawbacks. First, computing BN statistics across large batches has been shown to harm generalization [9]. Second, using global BN requires many aggregations across accelerators which incurs significant latency.
We investigated Group Normalization (GN) [60] and Weight Standardization (WS) [43] as alternatives to BN. We tested large batch training using 128 accelerator chips and a batch size of 4096. We found that GN alone does not scale to large batches; we observe a performance drop of 5.4% on ILSVRC-2012 top-1 accuracy compared to using BN in a ResNet-50x1. The addition of WS enables GN to scale to such large batches, even outperforming BN, see Table 4.
We are not only interested in upstream performance, but also how models trained with GN and WS transfer. We thus transferred models with different combinations of BN, GN, and WS pre-trained on ILSVRC-2012 to the 19 tasks defined by VTAB. The results in Table 5 indicate that the GN/WS combination transfers better than BN, so we use GN/WS in all BiT models.  [56]. This dataset appears to transfer less well than JFT-300M.While studying the effect of dataset size, [51] show good transfer performance when training on JFT-300M, despite reporting a large degree of noise (20% precision errors) in the labels. An even larger, noisily labelled dataset of 3.5B Instagram images is used in [36]. This increase in dataset size and an improved model architecture [62] lead to better results when transferring to ILSVRC-2012. We show that we can attain even better performance with ResNet using JFT-300M with appropriate adjustments presented in Section 2. The aforementioned papers focus on transfer to ImageNet classification, and COCO or VOC detection and segmentation. We show that transfer is also highly effective in the low data regime, and works well on the broader set of 19 tasks in VTAB [66].

Related Work
Specialized Representations Rather than pre-train generic representations, recent works have shown strong performance by training task-specific representations [63,38,61]. These papers condition on a particular task when training on a large support dataset. [63,61] train student networks on a large unlabelled support dataset using the predictions of a teacher network trained on the target task.
[38] compute importance weights on the a labelled support dataset by conditioning on the target dataset. They then train the representations on the re-weighted source data. Even though these approaches may lead to superior results, they require knowing the downstream dataset in advance and substantial computational resources for each downstream dataset.
Unsupervised and Semi-Supervised Representation learning Self-supervised methods have shown the ability to leverage unsupervised datasets to transfer to labelled tasks. For example, [13] show that unsupervised representations trained on 1B unlabelled Instagram images transfer comparably or better than supervised ILSVRC-2012 features. Semi-supervised learning exploits unsupervised data drawn from the same domain as the labelled data. [5,50] used semi-supervised learning to attain strong performance on CIFAR-10 and SVHN using only 40 or 250 labels. Recent works combine self-supervised and semi-supervised learning to attain good performance with fewer labels on Ima-geNet [65,17]. [66] study many representation learning algorithms (unsupervised, semi-supervised, and supervised) and evaluate their representation's ability to generalize to novel tasks, concluding that a combination of supervised and selfsupervised signals works best. However, all models were trained on ILSVRC-2012. We show that supervised pre-training on larger datasets continues to be an effective strategy.
Few-shot Learning Many strategies have been proposed to attain good performance when faced with novel classes and only a few examples per class. Metalearning or metric-learning techniques have been proposed to learn with few or no labels [59,49,52]. However, recent work has shown that a simple linear classifier on top of pre-trained representations or fine-tuning can attain similar or better performance [7,37]. The upstream pre-training and downstream few-shot learning are usually performed on the same domain, with disjoint class labels. In contrast, our goal is to find a generalist representation which works well when transferring to many downstream tasks.

Discussion
We revisit classical transfer learning, where a large pre-trained generalist model is fine-tuned to downstream tasks of interest. We provide a simple recipe which exploits large scale pre-training to yield good performance on all of these tasks. BiT uses a clean training and fine-tuning setup, with a small number of carefully selected components, to balance complexity and performance.
In Figure 8 and the Supplementary Material section D, we take a closer look at the remaining mistakes that BiT-L makes. In many cases, we see that these label/prediction mismatches are not true 'mistakes': the model's classification is valid, but it does not match the label. For example, the model may identify another prominent object when there are multiple objects in the image, or may provide an valid classification when the main entity has multiple attributes. There are also cases of label noise, where the model's prediction is a better fit than the ground-truth label. In a quantitative study, we found that around half of the model's mistakes on CIFAR-10 are due to ambiguity or label noise (see Figure 8, left), and in only 19.21% of the ILSVRC-2012 mistakes do human raters clearly agree with the label over the prediction. Overall, by inspecting these mistakes, we observe that performance on the standard vision benchmarks seems to approach a saturation point.
We therefore explore the effectiveness of transfer to two classes of more challenging tasks: classical image recognition tasks, but with very few labelled examples to adapt to the new domain, and VTAB, which contains more diverse tasks, such as spatial localization, tasks from simulated environments, and medical and satellite imaging tasks. These benchmarks are much further from saturation; while BiT-L performs well on them, there is still substantial room for further progress.

A Tuning hyperparameters for transfer
Throughout the paper we evaluate BiT using BiT-HyperRule. Here, we investigate whether BiT-L would benefit from additional computational budget for selecting fine-tuning hyperparameters. For this investigation we use VTAB-1k as it contains a diverse set of 19 tasks. For each task we fine-tune BiT-L 40 times using 800 training images. Each trial uses randomly sampled hyperparameters as described below. We select the best model for each dataset using the validation set with 200 images. The results are shown in Figure 9. Overall, we observe that VTAB-1k score saturates roughly after 20 trials and that further tuning results in overfitting on the validation split. This indicates that practitioners do not need to do very heavy tuning in order to find optimal parameters for their task.
After re-training BiT-L model with selected hyper-parameters using all union of training and validation splits (1000 images) we obtain the VTAB-1k score of 78.72%, an absolute improvement of 2.43% over 76.29% score obtained with BiT-HyperRule.
Our random search includes following hyperparameters with the following ranges and sampling strategies:  Figure 10 shows more results on the ObjectNet test set, with top-5 accuracy reported on the left and top-1 accuracy on the right. In Figure 10 (a), we first resize the shorter side of the image to 512 pixels and then take a 480×480 pixel sized central crop, similar to BiT-HyperRule.

B Full ObjectNet results
ObjectNet is a dataset collected in the real world, where multiple objects are present most of the time. A recent analysis shows that cropping out a single object from the cluttered scene could significantly improve performance [6]. In Figure 10 (b), we follow the same setup and report our models' performance on the cropped ObjectNet with a single object in each crop. We observe a solid improvement in performance in this setting.

C Duplicates and near-duplicates
In order to make sure that our results are not inflated due to overlap between upstream training and downstream test data, we run extensive de-duplication experiments. For training our flagship model, BiT-L, we remove all images from JFT-300M dataset that are duplicates and near-duplicates of test images of all our downstream datasets. In total, we removed less than 50 k images from the JFT-300M dataset. Interestingly, we did not observe any drastic difference by doing de-duplication, evidenced by comparing the first column of Table 1 (deduplicated upstream) and the first column of Table 7 (full upstream).
In another realistic setting, eventual downstream tasks are not known in advance. To better understand this setting, we also investigate how duplicates affect performance by removing them from the downstream test data after the upstream model has already been trained. The results of this experiment are shown in Table 7: "Full" is the accuracy on the original test set that contains nearduplicates, "Dedup" is the accuracy on the test set cleaned of near-duplicates, and "Dups" is the number of near-duplicates that have been removed from said test set. We observe that near-duplicates barely affect the results in all of our experiments. Note that near-duplicates between training and test sets have previously been reported by [51] for ILSVRC-2012, and by [3] for CIFAR.
In Figure 11, we present a few duplicates found between the ILSVRC-2012 training set and test splits of four standard downstream datasets.

D All of BiT-L's Mistakes
Here we take a closer look at the mistakes made by BiT-L 2 . Figure 8, we show all mistakes on CIFAR-10, as well as a representative selection of mistakes on ILSVRC-2012. Figures 12 and 13 again show all mistakes on the Pets and Flowers datasets, respectively. The first word always represents the model's prediction, while the second word represents the ground-truth label. The larger panels are best viewed on screen, where they can be magnified. 2 To be precise, the figures are obtained by an earlier version of our BiT-L model but which reaches almost the same accuracy. We did not re-run the figures and human evaluation with the latest model as they serve for illustration purposes and the models perform essentially the same, modulo a few flips.

E Object detection experiments
As discussed in the main text, for object detection evaluation we use the Reti-naNet model [33]. Our implementation is based on publicly available code 3 and uses standard hyper-parameters for training all detection models. We repeat training 5 times and report median performance. Specifically, we train all of our models for 30 epochs using a batch size of 256 with stochastic gradient descent, 0.08 initial learning rate, 0.9 momentum and 10 −4 weight decay. We decrease the initial learning rate by a factor of 10 at epochs number 16 and 22. We did try training for longer (60 epochs) and did not observe performance improvements. The input image resolution is 1024 × 1024. During training we use a data augmentation scheme as in [34]: random horizontal image flips and scale jittering. We set the classification loss parameters α to 0.25 and γ to 2.0, see [33] for the explanation of these parameters.

F Horizontal flipping and cropping for VTAB-1k tasks
When fine-tuning BiT models, we apply random horizontal flipping and cropping as image augmentations. However, these operations are not reasonable for certain VTAB tasks, where the semantic label (e.g. angle, location or object count) is not invariant to these operations.

G Robustness: Objects out-of-context
It has been shown that CNNs can lack robustness when classifying objects outof-context [4,42,47]. We investigate whether BiT not only improves classification accuracy, but also out-of-context robustness. For this, we create a dataset of foreground objects corresponding to ILSVRC-2012 classes pasted onto miscellaneous backgrounds (Fig. 14 left). We obtain images of foreground objects using OpenImages-v5 [28] segmentation masks. Figure 14 shows an example, and more are given in Figure 15. Sometime foreground objects are partially occluded, resulting in an additional challenge. We transfer BiT models pre-trained on various datasets to ILSVRC-2012 and see how they perform on this out-of-context dataset. In Figure 14 we can see that the performance of models pre-trained on ILSVRC-2012 saturates on the out-of-context dataset, whereas by using more data during pre-training of larger models, better performance on ILSVRC-2012 does translate to better out-ofcontext performance.
More qualitatively, when we look at the predictions of the models on outof-context data, we observe a tendency for BiT-L to confidently classify the foreground object regardless of the context, while ILSVRC-2012 models also predict objects absent from the image, but that could plausibly appear with the background. An example of this is shown in Figure 14 left.

G.1 Out of context dataset details
We generate this dataset by combining foreground objects extracted from Open-Images V5 [28] with backgrounds, licensed for reuse with modification, mined from search engine results.
Foreground objects In this study, we evaluate models that output predictions over ILSVRC-2012 classes. We therefore fine-tune BiT models on ILSVRC-2012 using BiT-HyperRule. We choose 20 classes from OpenImages that correspond to one such class or a subset thereof. These 20 classes cover a spectrum of different object types. We then extract foreground objects that belong to these classes from images in OpenImages using the provided segmentation masks. Note that this leads to some objects being partially occluded; however, humans can still easily recognize the objects, and we would like the same from our models.
Backgrounds We define a list of 41 backgrounds that cover a range of contexts such that (1) we have reasonable diversity, and (2) the objects we choose would not likely be seen in some of these backgrounds. We then collect a few examples of each background using a search engine, limiting to results licensed for reuse with modification. We take the largest square crop of the background from the top left corner.
We paste the extracted foreground objects onto the backgrounds. This results in a total of 3321 images in our dataset (81 foreground objects × 41 backgrounds). We fix the size of the objects such that the longest side corresponds to 80% of the width of the background; thus, the object is prominent in the image. Figure 15 shows more examples of out-of-context images from our dataset, contrasting the predictions given by a standard ResNet50 trained on ILSVRC-2012 from scratch and the predictions of BiT-L fine-tuned on ILSVRC-2012.

G.2 Image Attributions
In this section we provide attributions for images used to generate the examples from the out-of-context dataset. All images are licensed CC-BY-2.0 unless noted otherwise.