SoftSort: A Continuous Relaxation for the argsort Operator
  While sorting is an important procedure in computer science, the argsort
operator - which takes as input a vector and returns its sorting permutation -
has a discrete image and thus zero gradients almost everywhere. This prohibits
end-to-end, gradient-based learning of models that rely on the argsort
operator. A natural way to overcome this problem is to replace the argsort
operator with a continuous relaxation. Recent work has shown a number of ways
to do this, but the relaxations proposed so far are computationally complex. In
this work we propose a simple continuous relaxation for the argsort operator
which has the following qualities: it can be implemented in three lines of
code, achieves state-of-the-art performance, is easy to reason about
mathematically - substantially simplifying proofs - and is faster than
competing approaches. We open source the code to reproduce all of the
experiments and results.
