Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval
aamirshakir, tomaarsen, SeanLee97
We introduce the concept of embedding quantization and showcase their impact on retrieval speed, memory usage, disk space, and cost. We'll discuss how embeddings can be quantized in theory and in practice, after which we introduce a [demo](https://huggingface.co/spaces/sentence-transformers/quantized-retrieval) showing a real-life retrieval scenario of 41 million Wikipedia texts.


## Table of Contents

* [Why Embeddings?](#why-embeddings)
    + [Embeddings may struggle to scale](#embeddings-may-struggle-to-scale)
* [Improving scalability](#improving-scalability)
    + [Binary Quantization](#binary-quantization)
        - [Binary Quantization in Sentence Transformers](#binary-quantization-in-sentence-transformers)
        - [Binary Quantization in Vector Databases](#binary-quantization-in-vector-databases)
    + [Scalar (int8) Quantization](#scalar-int8-quantization)
        - [Scalar Quantization in Sentence Transformers](#scalar-quantization-in-sentence-transformers)
        - [Scalar Quantization in Vector Databases](#scalar-quantization-in-vector-databases)
    + [Combining Binary and Scalar Quantization](#combining-binary-and-scalar-quantization)
    + [Quantization Experiments](#quantization-experiments)
    + [Influence of Rescoring](#influence-of-rescoring)
        - [Binary Rescoring](#binary-rescoring)
        - [Scalar (Int8) Rescoring](#scalar-int8-rescoring)
        - [Retrieval Speed](#retrieval-speed)
    + [Performance Summarization](#performance-summarization)
    + [Demo](#demo)
    + [Try it yourself](#try-it-yourself)
    + [Future work:](#future-work)
    + [Acknowledgments](#acknowledgments)
    + [Citation](#citation)
    + [References](#references)

## Why Embeddings?

Embeddings are one of the most versatile tools in natural language processing, supporting a wide variety of settings and use cases. In essence, embeddings are numerical representations of more complex objects, like text, images, audio, etc. Specifically, the objects are represented as n-dimensional vectors. 

After transforming the complex objects, you can determine their similarity by calculating the similarity of the respective embeddings! This is crucial for many use cases: it serves as the backbone for recommendation systems, retrieval, one-shot or few-shot learning, outlier detection, similarity search, paraphrase detection, clustering, classification, and much more.

### Embeddings may struggle to scale

However, embeddings may be challenging to scale for production use cases, which leads to expensive solutions and high latencies. Currently, many state-of-the-art models produce embeddings with 1024 dimensions, each of which is encoded in `float32`, i.e., they require 4 bytes per dimension. To perform retrieval over 250 million vectors, you would therefore need around 1TB of memory!

The table below gives an overview of different models, dimension size, memory requirement, and costs. Costs are computed at an estimated $3.8 per GB/mo with `x2gd` instances on AWS.

| Embedding Dimension | Example Models                                                                                                                                          | 100M Embeddings         | 250M Embeddings | 1B Embeddings           |
|