language:en license:mit arxiv:2308.10335 region:us
---
# For reference on dataset card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/datasets-cards
name: RobustAPI
license: mit
size: 1208 questions
format: JSONL
language:
  - en
apis_used:
  - StackOverflow
  - GPT-3.5
  - GPT-4
  - Llama
  - PolyCoder
  - Vicuna
citation: >
    @misc{zhong2023chatgpt,
          title={Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation}, 
          author={Li Zhong and Zilong Wang},
          year={2023},
          eprint={2308.10335},
          archivePrefix={arXiv},
          primaryClass={cs.CL}
    }
---

# Dataset Card for RobustAPI

This dataset card provides details about the RobustAPI dataset introduced in the paper "Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation" (AAAI'24). The dataset comprises 1208 coding questions collected from StackOverflow related to 24 representative Java APIs. It includes information on API usage patterns and evaluations conducted on various large language models such as GPT-3.5, GPT-4, Llama, PolyCoder, and Vicuna.

## Dataset Details

### Dataset Description

RobustAPI contains coding questions from StackOverflow focusing on 24 Java APIs. It also includes summaries of API usage patterns and evaluations performed on popular large language models.

- **Curated by:** Li Zhong and Zilong Wang
- **License:** MIT License

### Dataset Sources

- **Paper:** [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://arxiv.org/abs/2308.10335)

## Uses

### Direct Use

RobustAPI can be used for evaluating the performance of large language models in code generation tasks and analyzing API usage patterns.

### Out-of-Scope Use

Misuse of the dataset for unintended purposes or malicious activities is discouraged.

## Dataset Structure

RobustAPI is provided in JSONL format. Each data point consists of a coding question along with corresponding API usage patterns and evaluations from various large language models.

## Dataset Creation

### Curation Rationale

The dataset was curated to investigate the feasibility of using large language models for code generation tasks and to study the robustness and reliability of such models compared to traditional sources like StackOverflow.

### Source Data

#### Data Collection and Processing

The data collection process involved gathering coding questions from StackOverflow related to 24 Java APIs. API usage patterns and evaluations were conducted using various large language models.

## Citation

To cite the dataset, please use the following:

**BibTeX:**

@misc{zhong2023chatgpt,
title={Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation},
author={Li Zhong and Zilong Wang},
year={2023},
eprint={2308.10335},
archivePrefix={arXiv},
primaryClass={cs.CL}
}

**APA:**

Zhong, L., & Wang, Z. (2023). Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation. arXiv preprint arXiv:2308.10335.
