Schr\"odinger's FP: Dynamic Adaptation of Floating-Point Containers for
  Deep Learning Training
  We introduce a software-hardware co-design approach to reduce memory traffic
and footprint during training with BFloat16 or FP32 boosting energy efficiency
and execution time performance. We introduce methods to dynamically adjust the
size and format of the floating-point containers used to store activations and
weights during training. The different value distributions lead us to different
approaches for exponents and mantissas. Gecko exploits the favourable exponent
distribution with a loss-less delta encoding approach to reduce the total
exponent footprint by up to $58\%$ in comparison to a 32 bit floating point
baseline. To content with the noisy mantissa distributions, we present two
lossy methods to eliminate as many as possible least significant bits while not
affecting accuracy. Quantum Mantissa, is a machine learning-first mantissa
compression method that taps on training's gradient descent algorithm to also
learn minimal mantissa bitlengths on a per-layer granularity, and obtain up to
$92\%$ reduction in total mantissa footprint. Alternatively, BitChop observes
changes in the loss function during training to adjust mantissa bit-length
network-wide yielding a reduction of $81\%$ in footprint. Schr\"{o}dinger's FP
implements hardware encoders/decoders that guided by Gecko/Quantum Mantissa or
Gecko/BitChop transparently encode/decode values when transferring to/from
off-chip memory boosting energy efficiency and reducing execution time.
