Practical Transferability Estimation for Image Classification Tasks
  Transferability estimation is an essential problem in transfer learning to
predict how good the performance is when transferring a source model (or source
task) to a target task. Recent analytical transferability metrics have been
widely used for source model selection and multi-task learning. A major
challenge is how to make transfereability estimation robust under the
cross-domain cross-task settings. The recently proposed OTCE score solves this
problem by considering both domain and task differences, with the help of
transfer experiences on auxiliary tasks, which causes an efficiency overhead.
In this work, we propose a practical transferability metric called JC-NCE score
that dramatically improves the robustness of the task difference estimation in
OTCE, thus removing the need for auxiliary tasks. Specifically, we build the
joint correspondences between source and target data via solving an optimal
transport problem with a ground cost considering both the sample distance and
label distance, and then compute the transferability score as the negative
conditional entropy of the matched labels. Extensive validations under the
intra-dataset and inter-dataset transfer settings demonstrate that our JC-NCE
score outperforms the auxiliary-task free version of OTCE for 7% and 12%,
respectively, and is also more robust than other existing transferability
metrics on average.
