task_categories:summarization language:en license:cc-by-4.0 size_categories:10M<n<100M doi:10.57967/hf/4327 region:us science agriculture academic
---
license: cc-by-4.0
task_categories:
- summarization
language:
- en
tags:
- science
- agriculture
- academic
size_categories:
- 10M<n<100M
---

# A Curated Research Corpus for Agricultural Advisory AI Applications
This dataset represents a comprehensive collection of 43,745 agricultural research publications from [CGIAR](https://cgiar.org/),
specifically processed and structured for Large Language Model (LLM) applications in agricultural advisory services. 
This dataset bridges the gap between advanced agricultural research and field-level advisory needs, 
drawing from CGIAR's extensive scientific knowledge base that has been used by both public and private extension services. 
Each document has been systematically processed using [GROBID](https://grobid.readthedocs.io/en/latest/Introduction/) to extract 
structured content while preserving critical scientific context, metadata, and domain-specific agricultural knowledge. Morever, chunking
methods that preserver the semantic coherence have been applied. More specifically, documents are split 
into chunks based on a fixed number of tokens and a portion of tokens at the end of each chunk 
overlaps with the beginning of the next chunk. This implementation Preserves contextual continuity between chunks, 
which improves the model's understanding of the document's flow and can lead to better predictions and is useful 
for tasks that rely on context spread over multiple chunks, such as question answering or summarization
([Chunking Methods](https://scio.atlassian.net/wiki/spaces/CiGi/pages/221675526/Chunking+methods)).
The corpus covers diverse agricultural topics including crop management, pest control, climate adaptation, and farming systems, 
with particular emphasis on small-scale producer contexts in low and middle-income countries. 
This machine-readable dataset is specifically curated to enhance the accuracy and contextual relevance of 
AI-generated agricultural advisories through Retrieval-Augmented Generation (RAG) frameworks, 
ensuring that advanced agricultural science can effectively benefit those at the heart of agriculture.

### Data Sources and RAG Pipeline
The dataset is sourced from [GARDIAN](https://gardian.bigdata.cgiar.org/), 
a comprehensive hub for agri-food data and publications. Utilizing its robust API, 
the GAIA-CIGI pipeline has systematically discovered and gathered all open-access reports and publications 
from the various CGIAR centers. Each document has been converted into a structured, machine-readable format using [GROBID](https://grobid.readthedocs.io/en/latest/Introduction/), 
a specialized tool for extracting the structure of scientific publications. A complete description of the system architecture can be found [here](https://scio.atlassian.net/wiki/spaces/CiGi/pages/45711361/Pipeline+Architecture)

### Document Structure
```
{
  "metadata": {
    "gardian_id": "",
    "source": "",
    "url": "",
    "id": ""
  },
  "keywords":["keywords"],
  "sieverID": "",
  "content": ""
}
```

### Property Description
<ol>
  <li>"metadata" (object, required): Contains information related to the document's metadata. 
    <ol>
      <li>"gardian_id" (string): an identifier for the document within the GARDIAN ecosystem.</li>
      <li>"source" (string): the source or origin of the document.</li>
      <li>"url" (string): the url of the downloaded document.</li>
      <li>"id" (string): internal identifier of the document generated by hashing the URL string.</li>      
    </ol>
  </li>
  <li>"keywords" (list of strings): the keyword list as obtained from origin index metadata.</li>
  <li>"sieverID" (string, required): internal identifier of the document.</li>
  <li>"content" (string): The useful textual content of the publication as retrieved using GROBID and PDFbox.</li>
</ol>

### Acknowledgement
This dataset was developed for the Generative AI for Agriculture (GAIA) project, supported by the Gates Foundation, in collaboration between [CGIAR](https://www.cgiar.org/)
and [SCiO](https://scio.systems/)