size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: repo
    dtype: string
  - name: docfile_name
    dtype: string
  - name: doc_type
    dtype: string
  - name: intent
    dtype: string
  - name: license
    dtype: string
  - name: path_to_docfile
    dtype: string
  - name: relevant_code_files
    sequence: string
  - name: relevant_code_dir
    dtype: string
  - name: target_text
    dtype: string
  - name: relevant_code_context
    dtype: string
  splits:
  - name: test
    num_bytes: 227163668
    num_examples: 216
  download_size: 30374553
  dataset_size: 227163668
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
---
# BenchName (Module summarization)

This is the benchmark for Module summarization task as part of the
 [BenchName benchmark](https://huggingface.co/collections/icmlbenchname/icml-25-benchname-679b57e38a389f054cdf33e9). 
The current version includes 216 manually curated text files describing different documentation of open-source permissive Python projects. 
The model is required to generate such description, given the relevant context code and the intent behind the documentation.
All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). The datapoints can be removed upon request.

## How-to

Load the data via [`load_dataset`](https://huggingface.co/docs/datasets/v2.14.3/en/package_reference/loading_methods#datasets.load_dataset):

    ```
    from datasets import load_dataset

    dataset = load_dataset("JetBrains-Research/lca-module-summarization")
    ```

## Datapoint Structure

Each example has the following fields:

|         **Field**           |             **Description**              |
|:---------------------------:|:----------------------------------------:|
|           `repo`            |            Name of the repository            |
|        `target_text`        |          Text of the target documentation file          |
|       `docfile_name`        |     Name of the file with target documentation    |
|          `intent`           |        One sentence description of what is expected in the documentation       |
|         `license`           |          License of the target repository          |
|    `relevant_code_files`    |      Paths to relevant code files (files that are mentioned in target documentation)      |
|     `relevant_code_dir`     |      Paths to relevant code directories  (directories that are mentioned in target documentation)      |
|      `path_to_docfile`      |      Path to file with documentation (path to the documentation file in source repository)    |
|   `relevant_code_context`   |          Relevant code context collected from relevant code files and directories          |


## Metric

To compare the predicted documentation and the ground truth documentation, we introduce the new metric based on LLM as an assessor. Our approach involves feeding the LLM with relevant code and two versions of documentation: the ground truth and the model-generated text. The LLM evaluates which documentation better explains and fits the code. To mitigate variance and potential ordering effects in model responses, we calculate the probability that the generated documentation is superior by averaging the results of two queries with the different order.

For more details about metric implementation, please refer to [our GitHub repository](https://anonymous.4open.science/r/icml-benchname-2025/).
