Where is the piece of code that computes dataset_info for each dataset uploaded to the hub?
As you may have noticed, each dataset uploaded to the huggingface hub is parsed automatically and the relevant informations are available at an URL like this:https://datasets-server.huggingface.co/info?dataset=lhoestq/demo1I’m interested in understanding how this parsing is performed, but I don’t know where to look for the source code.It doesn’t seem to be in the datasets library. Maybe it’s in the huggingface_hub library ? Or maybe it’s not open-source ?Any pointer would be appreciated
2024-11-15T00:34:33Z
I don’t know much about datasets, but there are a few projects on github that seem to be related.Not all of the services inside the HF server are open source, though. I guess that’s just how it has to be for security reasons.github.comGitHub - huggingface/evaluate: 🤗 Evaluate: A library for easily evaluating...🤗 Evaluate: A library for easily evaluating machine learning models and datasets.github.comGitHub - huggingface/dataset-viewer: Backend that powers the dataset viewer on Hugging...Backend that powers the dataset viewer on Hugging Face dataset pages through a public API.
2024-11-15T09:40:26Z
Thankks for the reply.It looks like it is somewhere in dataset-viewer indeed:github.comhuggingface/dataset-viewer/blob/main/docs/source/info.md?plain=1# Get dataset information

The dataset viewer provides an `/info` endpoint for exploring the general information about dataset, including such fields as description, citation, homepage, license and features.

The `/info` endpoint accepts two query parameters:

- `dataset`: the dataset name
- `config`: the subset name

<inferencesnippet>
<python>
```python
import requests
headers = {"Authorization": f"Bearer {API_TOKEN}"}
API_URL = "https://datasets-server.huggingface.co/info?dataset=ibm/duorc&config=SelfRC"
def query():
    response = requests.get(API_URL, headers=headers)
    return response.json()
data = query()
```This file has been truncated.show original
2024-11-15T09:56:25Z
It’s on the public part…