From Files to Chunks: Improving HF Storage Efficiency
jsulz, erinys
Hugging Face stores over [30 PB of models, datasets, and spaces](https://huggingface.co/spaces/xet-team/lfs-analysis) in [Git LFS repositories](https://huggingface.co/docs/hub/en/repositories-getting-started#requirements). Because Git stores and versions at the file level, any change to a file requires re-uploading the full asset – expensive operations when average Parquet and CSV files on the Hub range between 200-300 MB, average Safetensor files around 1 GB, and GGUF files can exceed 8 GB. Imagine modifying just a single line of metadata in a GGUF file and waiting for the multi-gigabyte file to upload; in addition to user time and transfer costs, Git LFS also then needs to save full versions of both files, bloating storage costs.

The plot below illustrates the growth of LFS storage in model, dataset, and space repositories on the Hub between March 2022 and September 2024:

<p align="center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/from-files-to-chunks/lfs-analysis-min.png" alt="Parquet Layout" width=90%>
</p>

Hugging Face's Xet team is taking a different approach to storage by storing files as chunks. By only transferring modified chunks, we can dramatically improve both storage efficiency and iteration speed while ensuring reliable access to evolving datasets and models. Here’s how it works.

## Content-Defined Chunking Foundations

The method that we use to chunk files is called content-defined chunking (CDC). Instead of treating a file as an indivisible unit, CDC breaks files down into variable-sized chunks, using the data to define boundaries. To compute the chunks, we apply a [rolling hash algorithm](https://en.wikipedia.org/wiki/Rolling_hash) that scans the file’s byte sequence.

Consider a file with the contents:

```bash
transformerstransformerstransformers
```

We’re using text for illustration, but this could be any sequence of bytes.

A rolling hash algorithm computes a hash over a sliding window of data. In this case, with a window of length 4 the hash would be computed first over `tran`, then `rans`, then `ansf` and so on until the end of the file.

Chunk boundaries are determined when a hash satisfies a predefined condition, such as:

```
hash(data) % 2^12 == 0
```

If the sequence `mers` produces a hash that meets this condition, the file will be split into three chunks:

```bash
transformers | transformers | transformers
```

The content of these chunks is hashed to create mapping between chunk hash and bytes and will eventually be stored in a content-addressed store (CAS). Since all three chunks are identical, we only store one chunk in the CAS for built-in deduplication. 🪄

## Insertions and Deletions

When the contents of a file change, CDC allows for fine-grained updates that make it robust to handling insertions and deletions. Let’s modify the file by inserting `super`, making the new file contents:

```bash
transformerstransformerssupertransformers
```

After applying the rolling hash again with the same boundary condition, the new chunks look like this:

```bash
 transformers | transformers | supertransformers
```

We do not need to save chunks we have seen before; they are already stored. However, `supertransformers` is a new chunk. Thus, the only cost of saving the updated version of this file is uploading and storing one new chunk.

To validate this optimization in the real world, we benchmarked our previous implementation of CDC-backed storage at XetHub against Git LFS and found a consistent 50% improvement in storage and transfer performance across three iterative development use cases. One example was the [CORD-19 dataset](https://ai2-semanticscholar-cord-19.s3-us-west-2.amazonaws.com/historical_releases.html), a collection of COVID-19 research papers curated between 2020 and 2022 with 50 incremental updates. The comparison between Xet-backed and Git LFS-backed repositories is summarized below:

| Metric                | Git LFS-backed Repository | Xet-backed Repository |
|