An Efficient Shared-memory Parallel Sinkhorn-Knopp Algorithm to Compute
  the Word Mover's Distance
  The Word Mover's Distance (WMD) is a metric that measures the semantic
dissimilarity between two text documents by computing the cost of moving all
words of a source/query document to the most similar words of a target document
optimally. Computing WMD between two documents is costly because it requires
solving an optimization problem that costs \(O(V^3log(V))\) where \(V\) is the
number of unique words in the document. Fortunately, the WMD can be framed as
the Earth Mover's Distance (EMD) (also known as the Optimal Transportation
Distance) for which it has been shown that the algorithmic complexity can be
reduced to \(O(V^2)\) by adding an entropy penalty to the optimization problem
and a similar idea can be adapted to compute WMD efficiently. Additionally, the
computation can be made highly parallel by computing WMD of a single query
document against multiple target documents at once (e.g., finding whether a
given tweet is similar to any other tweets happened in a day). In this paper,
we present a shared-memory parallel Sinkhorn-Knopp Algorithm to compute the WMD
of one document against many other documents by adopting the \(O(V^2)\) EMD
algorithm. We used algorithmic transformations to change the original dense
compute-heavy kernel to a sparse compute kernel and obtained \(67\times\)
speedup using \(96\) cores on the state-of-the-art of Intel\textregistered{}
4-sockets Cascade Lake machine w.r.t. its sequential run. Our parallel
algorithm is over \(700\times\) faster than the naive parallel python code that
internally uses optimized matrix library calls.
