EFloat: Entropy-coded Floating Point Format for Compressing Vector
  Embedding Models
  In a large class of deep learning models, including vector embedding models
such as word and database embeddings, we observe that floating point exponent
values cluster around a few unique values, permitting entropy based data
compression. Entropy coding compresses fixed-length values with variable-length
codes, encoding most probable values with fewer bits. We propose the EFloat
compressed floating point number format that uses a variable field boundary
between the exponent and significand fields. EFloat uses entropy coding on
exponent values and signs to minimize the average width of the exponent and
sign fields, while preserving the original FP32 exponent range unchanged. Saved
bits become part of the significand field increasing the EFloat numeric
precision by 4.3 bits on average compared to other reduced-precision floating
point formats. EFloat makes 8-bit and even smaller floats practical without
sacrificing the exponent range of a 32-bit floating point representation. We
currently use the EFloat format for saving memory capacity and bandwidth
consumption of large vector embedding models such as those used for database
embeddings. Using the RMS error as metric, we demonstrate that EFloat provides
higher accuracy than other floating point formats with equal bit budget. The
EF12 format with 12-bit budget has less end-to-end application error than the
16-bit BFloat16. EF16 with 16-bit budget has an RMS-error 17 to 35 times less
than BF16 RMS-error for a diverse set of embedding models. When making
similarity and dissimilarity queries, using the NDCG ranking metric, EFloat
matches the result quality of prior floating point representations with larger
bit budgets.
