size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
- config_name: generate_reranking_pairs
  features:
  - name: id
    dtype: string
  - name: anchor
    dtype: string
  - name: source
    dtype: string
  - name: positive
    dtype: string
  - name: negative
    dtype: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_generate_reranking_pairs
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_generate_reranking_pairs
      dtype: string
  - name: model_name
    dtype: string
  splits:
  - name: train
    num_bytes: 71357
    num_examples: 15
  download_size: 64430
  dataset_size: 71357
- config_name: generate_retrieval_pairs
  features:
  - name: id
    dtype: string
  - name: anchor
    dtype: string
  - name: source
    dtype: string
  - name: positive
    dtype: string
  - name: negative
    dtype: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_generate_retrieval_pairs
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_generate_retrieval_pairs
      dtype: string
  - name: model_name
    dtype: string
  splits:
  - name: train
    num_bytes: 72768
    num_examples: 15
  download_size: 59448
  dataset_size: 72768
configs:
- config_name: generate_reranking_pairs
  data_files:
  - split: train
    path: generate_reranking_pairs/train-*
- config_name: generate_retrieval_pairs
  data_files:
  - split: train
    path: generate_retrieval_pairs/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for example-retrieval-reranking-dataset2

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/aniruddha007/example-retrieval-reranking-dataset2/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/aniruddha007/example-retrieval-reranking-dataset2/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: generate_retrieval_pairs </summary><hr>

```json
{
    "anchor": ".rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.",
    "distilabel_metadata": {
        "raw_input_generate_retrieval_pairs": [
            {
                "content": "Your task is to generate a positive and a negative sentence given an anchor sentence. Take into account the context given. The positive sentence has to be a query for the anchor sentence, while the negative sentence is a \u0027hard negative\u0027 that meets the following criteria:\n- Uses similar keywords or phrases as the anchor sentence\n- Has a similar grammatical structure or syntax\n- Is not related to the anchor sentence, but could be mistaken for it\nTry to create a negative sentence that would be challenging for a model to distinguish from the positive sentence. You must output only two new sections: `## Positive` and `## Negative`.",
                "role": "system"
            },
            {
                "content": "## Context\n\n\nThe text is a chunk from technical Python SDK documentation for LangChain. LangChain is a framework designed to help AI engineers and domain experts build applications powered by large language models (LLMs). It supports the creation of high-quality datasets and automates workflows. The text may include explanatory prose, Python code snippets, and references to LangChain components.\n\n\n## Anchor\n\n.rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.\n",
                "role": "user"
            }
        ],
        "raw_output_generate_retrieval_pairs": "## Positive\nHow can I create a question answering application using LangChain and large language models (LLMs)?\n\n## Negative\nCan I develop a question answering application using LangChain but without large language models (LLMs)?"
    },
    "id": "9c04de564ed3-0",
    "model_name": "mistralai/Mistral-7B-Instruct-v0.3",
    "negative": "Can I develop a question answering application using LangChain but without large language models (LLMs)?",
    "positive": "How can I create a question answering application using LangChain and large language models (LLMs)?",
    "source": "https://langchain.readthedocs.io/en/latest/index.html"
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("aniruddha007/example-retrieval-reranking-dataset2", "generate_retrieval_pairs")
```


</details>

<details><summary> Configuration: generate_reranking_pairs </summary><hr>

```json
{
    "anchor": ".rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.",
    "distilabel_metadata": {
        "raw_input_generate_reranking_pairs": [
            {
                "content": "Your task is to generate a positive and a negative sentence given an anchor sentence. Take into account the context given. The positive sentence has to be semantically similar to the anchor sentence, while the negative sentence can use similar words but must not be related to the anchor sentence. You must output only two new sections: `## Positive` and `## Negative`.",
                "role": "system"
            },
            {
                "content": "## Context\n\n\nThe text is a chunk from technical Python SDK documentation for LangChain. LangChain is a framework designed to help AI engineers and domain experts build applications powered by large language models (LLMs). It supports the creation of high-quality datasets and automates workflows. The text may include explanatory prose, Python code snippets, and references to LangChain components.\n\n\n## Anchor\n\n.rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.\n",
                "role": "user"
            }
        ],
        "raw_output_generate_reranking_pairs": "## Positive\nLangChain offers a vast array of tools to help developers build powerful and innovative applications that leverage the capabilities of large language models (LLMs). By combining these LLMs with other sources of computation or knowledge, such as document loaders, Python REPLs, embedded, search engines, and more, developers can create applications that exceed the limitations of using LLMs in isolation.\n\n## Negative\nDespite the potential of large language models (LLMs) to revolutionize application development, simply using these LLMs without integrating them with other sources of computation or knowledge will not result in the creation of a truly powerful application. In fact, LangChain, a framework designed to assist in the development of such applications, is not focused on building applications using LLMs alone."
    },
    "id": "9c04de564ed3-0",
    "model_name": "mistralai/Mistral-7B-Instruct-v0.3",
    "negative": "Despite the potential of large language models (LLMs) to revolutionize application development, simply using these LLMs without integrating them with other sources of computation or knowledge will not result in the creation of a truly powerful application. In fact, LangChain, a framework designed to assist in the development of such applications, is not focused on building applications using LLMs alone.",
    "positive": "LangChain offers a vast array of tools to help developers build powerful and innovative applications that leverage the capabilities of large language models (LLMs). By combining these LLMs with other sources of computation or knowledge, such as document loaders, Python REPLs, embedded, search engines, and more, developers can create applications that exceed the limitations of using LLMs in isolation.",
    "source": "https://langchain.readthedocs.io/en/latest/index.html"
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("aniruddha007/example-retrieval-reranking-dataset2", "generate_reranking_pairs")
```


</details>




