Conformal retrofitting via Riemannian manifolds: distilling
  task-specific graphs into pretrained embeddings
  Pretrained (language) embeddings are versatile, task-agnostic feature
representations of entities, like words, that are central to many machine
learning applications. These representations can be enriched through
retrofitting, a class of methods that incorporate task-specific domain
knowledge encoded as a graph over a subset of these entities. However, existing
retrofitting algorithms face two limitations: they overfit the observed graph
by failing to represent relationships with missing entities; and they underfit
the observed graph by only learning embeddings in Euclidean manifolds, which
cannot faithfully represent even simple tree-structured or cyclic graphs. We
address these problems with two key contributions: (i) we propose a novel
regularizer, a conformality regularizer, that preserves local geometry from the
pretrained embeddings---enabling generalization to missing entities and (ii) a
new Riemannian feedforward layer that learns to map pre-trained embeddings onto
a non-Euclidean manifold that can better represent the entire graph. Through
experiments on WordNet, we demonstrate that the conformality regularizer
prevents even existing (Euclidean-only) methods from overfitting on link
prediction for missing entities, and---together with the Riemannian feedforward
layer---learns non-Euclidean embeddings that outperform them.
