Uncertainty Principle for Communication Compression in Distributed and
  Federated Learning and the Search for an Optimal Compressor
  In order to mitigate the high communication cost in distributed and federated
learning, various vector compression schemes, such as quantization,
sparsification and dithering, have become very popular. In designing a
compression method, one aims to communicate as few bits as possible, which
minimizes the cost per communication round, while at the same time attempting
to impart as little distortion (variance) to the communicated messages as
possible, which minimizes the adverse effect of the compression on the overall
number of communication rounds. However, intuitively, these two goals are
fundamentally in conflict: the more compression we allow, the more distorted
the messages become. We formalize this intuition and prove an {\em uncertainty
principle} for randomized compression operators, thus quantifying this
limitation mathematically, and {\em effectively providing asymptotically tight
lower bounds on what might be achievable with communication compression}.
Motivated by these developments, we call for the search for the optimal
compression operator. In an attempt to take a first step in this direction, we
consider an unbiased compression method inspired by the Kashin representation
of vectors, which we call {\em Kashin compression (KC)}. In contrast to all
previously proposed compression mechanisms, KC enjoys a {\em dimension
independent} variance bound for which we derive an explicit formula even in the
regime when only a few bits need to be communicate per each vector entry.
