Foreseeing the Benefits of Incidental Supervision
  Real-world applications often require improved models by leveraging a range
of cheap incidental supervision signals. These could include partial labels,
noisy labels, knowledge-based constraints, and cross-domain or cross-task
annotations -- all having statistical associations with gold annotations but
not exactly the same. However, we currently lack a principled way to measure
the benefits of these signals to a given target task, and the common practice
of evaluating these benefits is through exhaustive experiments with various
models and hyperparameters. This paper studies whether we can, in a single
framework, quantify the benefits of various types of incidental signals for a
given target task without going through combinatorial experiments. We propose a
unified PAC-Bayesian motivated informativeness measure, PABI, that
characterizes the uncertainty reduction provided by incidental supervision
signals. We demonstrate PABI's effectiveness by quantifying the value added by
various types of incidental signals to sequence tagging tasks. Experiments on
named entity recognition (NER) and question answering (QA) show that PABI's
predictions correlate well with learning performance, providing a promising way
to determine, ahead of learning, which supervision signals would be beneficial.
