An Inductive Bias for Distances: Neural Nets that Respect the Triangle
  Inequality
  Distances are pervasive in machine learning. They serve as similarity
measures, loss functions, and learning targets; it is said that a good distance
measure solves a task. When defining distances, the triangle inequality has
proven to be a useful constraint, both theoretically--to prove convergence and
optimality guarantees--and empirically--as an inductive bias. Deep metric
learning architectures that respect the triangle inequality rely, almost
exclusively, on Euclidean distance in the latent space. Though effective, this
fails to model two broad classes of subadditive distances, common in graphs and
reinforcement learning: asymmetric metrics, and metrics that cannot be embedded
into Euclidean space. To address these problems, we introduce novel
architectures that are guaranteed to satisfy the triangle inequality. We prove
our architectures universally approximate norm-induced metrics on
$\mathbb{R}^n$, and present a similar result for modified Input Convex Neural
Networks. We show that our architectures outperform existing metric approaches
when modeling graph distances and have a better inductive bias than non-metric
approaches when training data is limited in the multi-goal reinforcement
learning setting.
