Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision
  Foundation models offer an exciting new paradigm for constructing models with
out-of-the-box embeddings and a few labeled examples. However, it is not clear
how to best apply foundation models without labeled data. A potential approach
is to fuse foundation models with weak supervision frameworks, which use weak
label sources -- pre-trained models, heuristics, crowd-workers -- to construct
pseudolabels. The challenge is building a combination that best exploits the
signal available in both foundation models and weak sources. We propose Liger,
a combination that uses foundation model embeddings to improve two crucial
elements of existing weak supervision techniques. First, we produce finer
estimates of weak source quality by partitioning the embedding space and
learning per-part source accuracies. Second, we improve source coverage by
extending source votes in embedding space. Despite the black-box nature of
foundation models, we prove results characterizing how our approach improves
performance and show that lift scales with the smoothness of label
distributions in embedding space. On six benchmark NLP and video tasks, Liger
outperforms vanilla weak supervision by 14.1 points, weakly-supervised kNN and
adapters by 11.8 points, and kNN and adapters supervised by traditional hand
labels by 7.2 points.
