size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars library:distilabel region:us synthetic distilabel rlaif
---
size_categories: n<1K
dataset_info:
- config_name: generate_reranking_pairs
  features:
  - name: id
    dtype: string
  - name: anchor
    dtype: string
  - name: source
    dtype: string
  - name: positive
    dtype: string
  - name: negative
    dtype: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_generate_reranking_pairs
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_generate_reranking_pairs
      dtype: string
    - name: statistics_generate_reranking_pairs
      struct:
      - name: input_tokens
        dtype: int64
      - name: output_tokens
        dtype: int64
  - name: model_name
    dtype: string
  splits:
  - name: train
    num_bytes: 71934
    num_examples: 15
  download_size: 66411
  dataset_size: 71934
- config_name: generate_retrieval_pairs
  features:
  - name: id
    dtype: string
  - name: anchor
    dtype: string
  - name: source
    dtype: string
  - name: positive
    dtype: string
  - name: negative
    dtype: string
  - name: distilabel_metadata
    struct:
    - name: raw_input_generate_retrieval_pairs
      list:
      - name: content
        dtype: string
      - name: role
        dtype: string
    - name: raw_output_generate_retrieval_pairs
      dtype: string
    - name: statistics_generate_retrieval_pairs
      struct:
      - name: input_tokens
        dtype: int64
      - name: output_tokens
        dtype: int64
  - name: model_name
    dtype: string
  splits:
  - name: train
    num_bytes: 74335
    num_examples: 15
  download_size: 66290
  dataset_size: 74335
configs:
- config_name: generate_reranking_pairs
  data_files:
  - split: train
    path: generate_reranking_pairs/train-*
- config_name: generate_retrieval_pairs
  data_files:
  - split: train
    path: generate_retrieval_pairs/train-*
tags:
- synthetic
- distilabel
- rlaif
---

<p align="left">
  <a href="https://github.com/argilla-io/distilabel">
    <img src="https://raw.githubusercontent.com/argilla-io/distilabel/main/docs/assets/distilabel-badge-light.png" alt="Built with Distilabel" width="200" height="32"/>
  </a>
</p>

# Dataset Card for langchain-docs-retrieval-pairs

This dataset has been created with [distilabel](https://distilabel.argilla.io/).



## Dataset Summary

This dataset contains a `pipeline.yaml` which can be used to reproduce the pipeline that generated it in distilabel using the `distilabel` CLI:

```console
distilabel pipeline run --config "https://huggingface.co/datasets/sud-962081/langchain-docs-retrieval-pairs/raw/main/pipeline.yaml"
```

or explore the configuration:

```console
distilabel pipeline info --config "https://huggingface.co/datasets/sud-962081/langchain-docs-retrieval-pairs/raw/main/pipeline.yaml"
```

## Dataset structure

The examples have the following structure per configuration:


<details><summary> Configuration: generate_reranking_pairs </summary><hr>

```json
{
    "anchor": ".rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.",
    "distilabel_metadata": {
        "raw_input_generate_reranking_pairs": [
            {
                "content": "Your task is to generate a positive and a negative sentence given an anchor sentence. Take into account the context given. The positive sentence has to be semantically similar to the anchor sentence, while the negative sentence can use similar words but must not be related to the anchor sentence. You must output only two new sections: `## Positive` and `## Negative`.",
                "role": "system"
            },
            {
                "content": "## Context\n\n\nThe text is a chunk from technical Python SDK documentation for LangChain.\nLangChain is a framework designed to help AI engineers and domain experts build\napplications powered by large language models (LLMs). It supports the creation of high-quality datasets and automates workflows.\nThe text may include explanatory prose, Python code snippets, and references to LangChain components\n\n\n## Anchor\n\n.rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.\n",
                "role": "user"
            }
        ],
        "raw_output_generate_reranking_pairs": "## Positive\nLangChain\u0027s extensive modules and support for various language models (LLMs) not only empower developers to build innovative applications, but also facilitate the integration of these models with other computational and knowledge sources, enhancing their potential for creating powerful and efficient applications.\n\n## Negative\nWhile LangChain does provide a wealth of resources for building applications with language models (LLMs), it does not offer tools for developing traditional desktop or mobile applications, focusing primarily on integrating LLMs with other sources of computation or knowledge.",
        "statistics_generate_reranking_pairs": {
            "input_tokens": 609,
            "output_tokens": 110
        }
    },
    "id": "9c04de564ed3-0",
    "model_name": "mistralai/Mistral-7B-Instruct-v0.3",
    "negative": "While LangChain does provide a wealth of resources for building applications with language models (LLMs), it does not offer tools for developing traditional desktop or mobile applications, focusing primarily on integrating LLMs with other sources of computation or knowledge.",
    "positive": "LangChain\u0027s extensive modules and support for various language models (LLMs) not only empower developers to build innovative applications, but also facilitate the integration of these models with other computational and knowledge sources, enhancing their potential for creating powerful and efficient applications.",
    "source": "https://langchain.readthedocs.io/en/latest/index.html"
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("sud-962081/langchain-docs-retrieval-pairs", "generate_reranking_pairs")
```


</details>

<details><summary> Configuration: generate_retrieval_pairs </summary><hr>

```json
{
    "anchor": ".rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.",
    "distilabel_metadata": {
        "raw_input_generate_retrieval_pairs": [
            {
                "content": "Your task is to generate a positive and a negative sentence given an anchor sentence. Take into account the context given. The positive sentence has to be a query for the anchor sentence, while the negative sentence is a \u0027hard negative\u0027 that meets the following criteria:\n- Uses similar keywords or phrases as the anchor sentence\n- Has a similar grammatical structure or syntax\n- Is not related to the anchor sentence, but could be mistaken for it\nTry to create a negative sentence that would be challenging for a model to distinguish from the positive sentence. You must output only two new sections: `## Positive` and `## Negative`.",
                "role": "system"
            },
            {
                "content": "## Context\n\n\nThe text is a chunk from technical Python SDK documentation for LangChain.\nLangChain is a framework designed to help AI engineers and domain experts build\napplications powered by large language models (LLMs). It supports the creation of high-quality datasets and automates workflows.\nThe text may include explanatory prose, Python code snippets, and references to LangChain components\n\n\n## Anchor\n\n.rst\n.pdf\nWelcome to LangChain\n Contents \nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\u2753 Question Answering over specific documents\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\ud83d\udcac Chatbots\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\ud83e\udd16 Agents\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\nGetting Started Documentation\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.\n",
                "role": "user"
            }
        ],
        "raw_output_generate_retrieval_pairs": "## Positive\nWhat are some common examples of applications that can be built using LangChain, a framework for combining large language models (LLMs) with other sources of computation or knowledge?\n\n## Negative\nWhile LangChain is a powerful tool for building applications that leverage large language models (LLMs), it\u0027s important to note that it\u0027s not designed to assist in the development of applications that are purely based on small language models (SLMs).",
        "statistics_generate_retrieval_pairs": {
            "input_tokens": 665,
            "output_tokens": 95
        }
    },
    "id": "9c04de564ed3-0",
    "model_name": "mistralai/Mistral-7B-Instruct-v0.3",
    "negative": "While LangChain is a powerful tool for building applications that leverage large language models (LLMs), it\u0027s important to note that it\u0027s not designed to assist in the development of applications that are purely based on small language models (SLMs).",
    "positive": "What are some common examples of applications that can be built using LangChain, a framework for combining large language models (LLMs) with other sources of computation or knowledge?",
    "source": "https://langchain.readthedocs.io/en/latest/index.html"
}
```

This subset can be loaded as:

```python
from datasets import load_dataset

ds = load_dataset("sud-962081/langchain-docs-retrieval-pairs", "generate_retrieval_pairs")
```


</details>




