Active WeaSuL: Improving Weak Supervision with Active Learning
  The availability of labelled data is one of the main limitations in machine
learning. We can alleviate this using weak supervision: a framework that uses
expert-defined rules $\boldsymbol{\lambda}$ to estimate probabilistic labels
$p(y|\boldsymbol{\lambda})$ for the entire data set. These rules, however, are
dependent on what experts know about the problem, and hence may be inaccurate
or may fail to capture important parts of the problem-space. To mitigate this,
we propose Active WeaSuL: an approach that incorporates active learning into
weak supervision. In Active WeaSuL, experts do not only define rules, but they
also iteratively provide the true label for a small set of points where the
weak supervision model is most likely to be mistaken, which are then used to
better estimate the probabilistic labels. In this way, the weak labels provide
a warm start, which active learning then improves upon. We make two
contributions: 1) a modification of the weak supervision loss function, such
that the expert-labelled data inform and improve the combination of weak
labels; and 2) the maxKL divergence sampling strategy, which determines for
which data points expert labelling is most beneficial. Our experiments show
that when the budget for labelling data is limited (e.g. $\leq 60$ data
points), Active WeaSuL outperforms weak supervision, active learning, and
competing strategies, with only a handful of labelled data points. This makes
Active WeaSuL ideal for situations where obtaining labelled data is difficult.
