Dynamic Knowledge Distillation for Black-box Hypothesis Transfer
  Learning
  In real world applications like healthcare, it is usually difficult to build
a machine learning prediction model that works universally well across
different institutions. At the same time, the available model is often
proprietary, i.e., neither the model parameter nor the data set used for model
training is accessible. In consequence, leveraging the knowledge hidden in the
available model (aka. the hypothesis) and adapting it to a local data set
becomes extremely challenging. Motivated by this situation, in this paper we
aim to address such a specific case within the hypothesis transfer learning
framework, in which 1) the source hypothesis is a black-box model and 2) the
source domain data is unavailable. In particular, we introduce a novel
algorithm called dynamic knowledge distillation for hypothesis transfer
learning (dkdHTL). In this method, we use knowledge distillation with
instance-wise weighting mechanism to adaptively transfer the "dark" knowledge
from the source hypothesis to the target domain.The weighting coefficients of
the distillation loss and the standard loss are determined by the consistency
between the predicted probability of the source hypothesis and the target
ground-truth label.Empirical results on both transfer learning benchmark
datasets and a healthcare dataset demonstrate the effectiveness of our method.
