Share your open ML datasets on Hugging Face Hub!
davanstrien, cfahlgren1, lhoestq, erinys
Hugging Face Hub makes it seamless to host and share datasets, trusted by many leading research institutions, companies, and government agencies, including [Nvidia](https://huggingface.co/nvidia), [Google](https://huggingface.co/google), [Stanford](https://huggingface.co/stanfordnlp), [NASA](https://huggingface.co/ibm-nasa-geospatial), [THUDM](https://huggingface.co/THUDM) and [Barcelona Supercomputing Center](https://huggingface.co/BSC-LT).
 
By hosting a dataset on the Hugging Face Hub, you get instant access to features that can maximize your work's impact:

- [Generous Limits](#generous-limits)
- [Dataset Viewer](#dataset-viewer)
- [Third Party Library Support](#third-party-library-support)
- [SQL Console](#sql-console)
- [Security](#security)
- [Reach and Visibility](#reach-and-visibility)

##  Generous Limits

### Support for large datasets

The Hub can host terabyte-scale datasets, with high [per-file and per-repository limits](https://huggingface.co/docs/hub/en/repositories-recommendations). If you have data to share, the Hugging Face datasets team can help suggest the best format for uploading your data for community usage. 
The [🤗 Datasets library](https://huggingface.co/docs/datasets/index) makes it easy to upload and download your files, or even create a dataset from scratch. 🤗 Datasets also enables dataset streaming , making it possible to work with large datasets without needing to download the entire thing. This can be invaluable to allow researchers with less computational resources to work with your datasets, or to select small portions of a huge dataset for testing, development or prototyping.


<p align="center"> 
 <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/researcher-dataset-sharing/filesize.png" alt="Screenshot of the file size information for a dataset"><br> 
<em>The Hugging Face Hub can host the large datasets often created for machine learning research.</em> 
 </p> 

_Note: The [Xet team](https://huggingface.co/xet-team) is currently working on a backend update that will increase per-file limits from the current 50 GB to 500 GB while also improving storage and transfer efficiency._ 

## Dataset Viewer

Beyond just hosting your data, the Hub provides powerful tools for exploration. With the Datasets Viewer, users can explore and interact with datasets hosted on the Hub directly in their browser. This provides an easy way for others to view and explore your data without downloading it first.

Hugging Face datasets supports many different modalities (audio, images, video, etc.) and file formats (CSV, JSON, Parquet, etc.), and compression formats (Gzip, Zip, etc.). Check out the [Datasets File Formats](https://huggingface.co/docs/hub/en/datasets-adding#file-formats) page for more details.

<p align="center"> 
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/infinity-instruct.png" alt="Screenshot of the Datasets Viewer"><br> 
<em>The Dataset Viewer for the Infinity-Instruct dataset.</em> 
</p> 

The Datasets Viewer also includes a few features which make it easier to explore a dataset.

### Full Text Search

Built-in Full Text Search is one of the most powerful features of the Datasets Viewer. Any text columns in a dataset immediately become searchable.

The Arxiver dataset contains 63.4k rows of arXiv research papers converted to Markdown. By using Full Text Search, it's easy to find the papers containing a specific author such as Ilya Sutskever below.

<iframe
  src="https://huggingface.co/datasets/neuralwork/arxiver/embed/viewer/default/train?q=ilya+sutskever"
  frameborder="0"
  width="100%"
  height="560px"
></iframe>

### Sorting

The Datasets Viewer allows you to sort the dataset by clicking on the column headers. This makes it easy to find the most relevant examples in a dataset.

Below is an example of a dataset sorted by the `helpfulness` column in descending order for the [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2) dataset.

<iframe
  src="https://huggingface.co/datasets/nvidia/HelpSteer2/embed/viewer/default/train?sort[column]=helpfulness&sort[direction]=desc"
  frameborder="0"
  width="100%"
  height="560px"
></iframe>

## Third Party Library Support

Hugging Face is fortunate to have third party integrations with the leading open source data tools. By hosting a dataset on the Hub, it instantly makes the dataset compatible with the tools users are most familiar with.

Here are some of the libraries Hugging Face supports out of the box:

| Library | Description | Monthly PyPi Downloads (2024) |
| :