Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting
  Model Hubs
  Model hubs with many pre-trained models (PTMs) have been a cornerstone in
deep learning. Although built at a high cost, they remain
\emph{under-exploited}: practitioners usually pick one PTM from the provided
model hub by popularity and then fine-tune the PTM to solve the target task.
This na\"ive but common practice poses two obstacles to sufficient exploitation
of pre-trained model hubs: (1) the PTM selection by popularity has no
optimality guarantee; (2) only one PTM is used while the rest PTMs are ignored.
Ideally, to exploit pre-trained model hubs maximally, trying all combinations
of PTMs and extensively fine-tuning each PTM combination are required, which
incurs exponential combinations and an unaffordable computational budget. In
this paper, we propose a new paradigm of exploiting model hubs by ranking and
tuning pre-trained models: (1) Our conference paper~\citep{you_logme:_2021}
proposed LogME to estimate the maximum value of label evidence given features
extracted by pre-trained models, which can rank all the PTMs in a model hub for
various types of PTMs and tasks \emph{before fine-tuning}. (2) The best ranked
PTM can be fine-tuned and deployed if we have no preference for the model's
architecture, or the target PTM can be tuned by top-K ranked PTMs via the
proposed B-Tuning algorithm. The ranking part is based on the conference paper,
and we complete its theoretical analyses in this paper, including the
convergence proof of the heuristic evidence maximization procedure and the
influence of feature dimension. The tuning part introduces a novel Bayesian
Tuning (B-Tuning) method for tuning multiple PTMs, which surpasses specialized
methods designed for tuning homogeneous PTMs and sets up a new state of the art
for tuning heterogeneous PTMs. The new paradigm of exploiting PTM hubs can be
interesting to a large audience across the machine learning community.
