task_categories:question-answering language:en license:mit size_categories:n<1K region:us
---
license: mit
task_categories:
- question-answering
language:
- en
size_categories:
- n<1K
---
# Dataset Card for Dataset Name

<!-- Provide a quick summary of the dataset. -->

<!-- This dataset card aims to be a base template for new datasets. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md?plain=1).-->
This dataset consists of three novel reasoning datasets.  Each dataset contains a response prompt (a
question to ask a user or an LLM) and a ground truth answer. 

## Dataset Details
This dataset consists of three tasks: counting, multihop, and factoid.  Each row of the dataset consists of two columns: the prompt and the answer.  
The counting dataset evaluates the ability to count dates.  The multihop or RAG task evaluates the ability to retrieve information about open hours for CVS stores in California.
The factoid evaluates the ability to retrieve information when there might not be an answer.  

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->
-   **Dataset 1: Arithmetic (Counting)**:
-   This dataset tests the ability to count days in a particular country
with multiple arrival and departure dates. The ground truth answer is
a single integer number. The dataset is generated for a set of 100
arrival and departure dates.
-   **Dataset 2: Multihop (RAGs)**: This dataset tests the
ability to perform information retrieval to determine if there are
24/7 CVS pharmacies in a specified city. The ground truth answer is a
single integer number indicating the number of 24-hour pharmacies,
which could be 0. The dataset is generated for the 387 cities in
California with a CVS store.  
-   **Dataset 3: Factoid**:  This
dataset tests the ability to retrieve facts where there may not be an
answer. This dataset asks about the national sport for a specified
country, which may not exist. The ground truth answer is a string
indicating the national sport or "Nothing" if the national sport does
not exist. This dataset is generated for the 249 countries in the
world.


- **Curated by:** Nikhil Wani and Leilani Gilpin
- **Funded by:** Funding for this work was provided by Underwriters Laboratories Inc. through the Center for Advancing Safety of Machine Intelligence.
<!-- - **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed] --> 

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

<!-- - **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed] -->

## Uses

<!-- Address questions around how the dataset is intended to be used. -->
This dataset is designed to evaluate AI models across three key capabilities: arithmetic reasoning, multistep information retrieval, and fact-based knowledge queries. The arithmetic dataset involves calculating the total days spent in the U.S. based on simulated travel itineraries, testing logical date-based reasoning. The multihop dataset assesses a model’s ability to retrieve and aggregate information, such as identifying 24/7 CVS pharmacies in California cities through multiple steps. The factoid dataset evaluates the ability to retrieve factual information, including handling cases where no definitive answer exists, like determining a country's national sport. These datasets collectively serve as a benchmark for testing and improving AI systems’ reasoning, retrieval, and knowledge-handling skills.

### Direct Use

<!-- This section describes suitable use cases for the dataset. -->
This dataset is suited for benchmarking and improving AI systems in
arithmetic reasoning, multistep information retrieval, and fact-based
knowledge tasks. It can be used to train or evaluate models for
the ability to handle uncertainty in tasks. It supports the development of retrieval-augmented systems and
knowledge-based applications, including trivia systems and tools that
handle "null answer" scenarios. 

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->
This dataset is not suitable for tasks outside its specific domains of
arithmetic reasoning, information retrieval, and fact-based question
answering. It unsuitable for training or evaluating models
on open-domain reasoning, ethical decision-making, or multimodal
applications involving images or audio. 

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->
The dataset consists of two columns, the prompt and the true answer.  

## Dataset Creation
-   **Dataset 1: Arithmetic (Counting)**: This dataset tests the ability
    to count days in a particular country with multiple arrival and
    departure dates. The ground truth answer is a single integer number
    representing the number of days stayed in the US. This dataset is
    generated by creating four random dates to simulate a travel
    itinerary:
    
    -   Start Date: A randomly selected date within the current year,
        marking the start of the trip.
    -   First Departure: Set by adding a random number (up to 10) of days
        to the start date.
    -   Second Arrival: Fixed to 40 days after the start date, ensuring it
        occurs after the first departure.
    -   Second Departure: Determined by adding a random number (up to 10)
        of days to the second arrival date.
    
    The task is to calculate how many days were spent in the US, which
    is determined by summing by summing the difference between the first
    departure and start date, the difference between the second
    departure and second arrival, plus 2 (to account for fencepost errors.
-   **Dataset 2: Multihop (RAGs)**: This dataset tests the ability to
    perform information retrieval to determine if there are 24/7 CVS
    pharmacies in a specified city. The ground truth answer is a single
    integer number indicating the number of 24-hour pharmacies, from 0
    to 2. To generate the dataset, a list of 387 California cities with
    CVS stores was obtained by scraping the CVS store locator page:
    <https://www.cvs.com/store-locator/cvs-pharmacy-locations/California/>. For
    each city, we determined the number of 24-hour CVS pharmacies by
    scraping the pharmacy locations page for that city. Using
    BeautifulSoup, we checked each location's opening hours, adding one
    to the city count if a pharmacy operates 24/7.
-   **Dataset 3: Factoid**: This dataset tests the ability to retrieve
    facts where there may not be an answer. This dataset asks about the
    national sport for a specified country, which may not exist. The
    ground truth answer is a string indicating the national sport or
    "Nothing" if the national sport does not exist. This dataset is
    generated for the 249 countries in the world.  To generate this
    dataset, we obtained a list of the 249 countries of the world using
    `pycountry.countries`.  Then, we used the list of national sports
    from Wikipedia: <https://en.wikipedia.org/wiki/National_sport> and
    used BeautifulSoup to aggregate the national sport for each country
    (if it exists).  For countries without a designated national sport,
    the value is set to "Nothing."

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->
The curation rationale for this dataset is centered on creating
targeted benchmarks to evaluate LLMs on uncertain tasks.Each dataset
is designed to simulate realistic scenarios: the arithmetic dataset
emphasizes logical reasoning with date-based calculations, reflecting
common tasks like travel planning. The multihop dataset requires
retrieving and aggregating structured information across multiple
steps. The factoid dataset tests knowledge retrieval and handling of
incomplete information, such as cases where answers may not exist. 

### Source Data
The source data comes from Wikipedia, and the CVS website.  
<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

#### Data Collection and Processing
Beautiful Soup was used to scrape information from the source data.
<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->


#### Who are the source data producers? 
The source data for this dataset comes from publicly available
resources and synthetic generation. The arithmetic dataset uses
randomized date logic without external data. The multihop dataset
relies on CVS's official store locator and pharmacy location pages. The
factoid dataset draws from the `pycountry.countries` Python library for
a list of countries and Wikipedia's national sports page for factual
information. 

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

<!-- [More Information Needed] -->

<!-- ### Annotations [optional]

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

<!-- #### Annotation process

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

<!-- [More Information Needed] -->

<!-- #### Who are the annotators?

<!-- This section describes the people or systems who created the annotations. -->

<!-- [More Information Needed] -->

#### Personal and Sensitive Information
There is no personal nor sensitive information.

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
Since the data is synthetically generated or scraped from specific
online sources, it may not fully capture the diversity of real-world
scenarios. For example, the arithmetic dataset's travel itineraries
are based on random date generation, which may not reflect real-world
travel patterns or demographic factors that could affect travel
behavior. The multihop dataset is centered on CVS stores in California,
may introduce regional bia. The factoid dataset relies on Wikipedia for
national sports data, which may be incomplete or biased, as not all
countries have a nationally recognized sport. 

<!-- ### Recommendations  -->

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

<!-- Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations. -->

<!-- ## Citation [optional] -->

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

<!-- **BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional] -->

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

<!-- [More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed]

## Dataset Card Contact

[More Information Needed] -->