size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: repo_full_name
    dtype: string
  - name: repo_name
    dtype: string
  - name: repo_owner
    dtype: string
  - name: instruction
    dtype: string
  - name: reference
    dtype: string
  - name: clean_reference
    dtype: string
  - name: path_to_reference_file
    dtype: string
  - name: path_to_examples_folder
    dtype: string
  - name: n_unique_apis
    dtype: int64
  - name: unique_apis
    sequence: string
  - name: project_defined_elements
    sequence: string
  - name: api_calls
    sequence: string
  - name: internal_apis
    sequence: string
  splits:
  - name: test
    num_bytes: 8652521
    num_examples: 150
  download_size: 4574177
  dataset_size: 8652521
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
---

# BenchName (Library-based code generation)

This is the benchmark for Library-based code generation task as part of the
 [BenchName benchmark](https://huggingface.co/collections/icmlbenchname/icml-25-benchname-679b57e38a389f054cdf33e9).

The current version includes 150 manually curated instructions asking the model to generate Python code using a particular library. 
The samples come from 62 Python repositories. 
All the samples in the dataset are based on reference example programs written by authors of the respective libraries. 
All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). 
The datapoints can be removed upon request.

## How-to

Load the data via [`load_dataset`](https://huggingface.co/docs/datasets/v2.14.3/en/package_reference/loading_methods#datasets.load_dataset):

```
from datasets import load_dataset

dataset = load_dataset("icmlbenchname/library-based-code-generation", split="test")
```

Note that all the data we have is considered to be in the test split.

## Dataset Structure

### Samples structure

The samples in the dataset have the following structure:


|          **Field**          |                                       **Description**                                        |
|:---------------------------:|:--------------------------------------------------------------------------------------------:|
|  `repo_full_name`           |  Concatenated repository name and owner                                                      |
|  `repo_name`                |  Library repository name                                                                     |
|  `repo_owner`               |  Library repository owner                                                                    |
|  `instruction`              |  Task for code generation                                                                    |
|  `reference`                |  Reference program written by the library authors                                            |
|  `clean_reference`          |  Reference program with comments removed                                                     |
|  `path_to_reference_file`   |  Path to the reference in the repository (removed in archive to prevent data leakages)       |
|  `path_to_examples_folder`  |  Path to the examples folder in the repository (removed in archive to prevent data leakages) |
|  `n_unique_apis`            |  Number of calls to library-specific APIs in the reference program                           |
|  `unique_apis`              |  List of of calls to library-specific APIs in the reference program                          |
|  `project_defined_elements` |  All class and method names in the repository                                                |
|  `api_calls`                |  All API calls (including referencing other libraries) in the reference program              |
|  `internal_apis`            |  All API calls (including appearing in other libraries) in the reference program             |
