Effect of Pre-Training Scale on Intra- and Inter-Domain Full and
  Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
  Transfer learning aims to exploit pre-trained models for more efficient
follow-up training on wide range of downstream tasks and datasets, enabling
successful training also on small data. Recently, strong improvement was shown
for transfer learning and model generalization when increasing model, data and
compute budget scale in the pre-training. To compare effect of scale both in
intra- and inter-domain full and few-shot transfer, in this study we combine
for the first time large openly available medical X-Ray chest imaging datasets
to reach a dataset scale comparable to ImageNet-1k. We then conduct
pre-training and transfer to different natural or medical targets while varying
network size and source data scale and domain, being either large natural
(ImageNet-1k/21k) or large medical chest X-Ray datasets. We observe strong
improvement due to larger pre-training scale for intra-domain natural-natural
and medical-medical transfer. For inter-domain natural-medical transfer, we
find improvements due to larger pre-training scale on larger X-Ray targets in
full shot regime, while for smaller targets and for few-shot regime the
improvement is not visible. Remarkably, large networks pre-trained on very
large natural ImageNet-21k are as good or better than networks pre-trained on
largest available medical X-Ray data when performing transfer to large X-Ray
targets. We conclude that high quality models for inter-domain transfer can be
also obtained by substantially increasing scale of model and generic natural
source data, removing necessity for large domain-specific medical source data
in the pre-training. Code is available at:
\url{https://github.com/SLAMPAI/large-scale-pretraining-transfer}}
