license:mit size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
---

This is an Encephalitis Title, Abstracts, and Search Queries Dataset

This dataset contains pairs of encephalitis title/abstracts and related search queries. The Title is first sentence in column called Title_Abstract. 

The search queries are highly relevant to the abstracts, but use different keywords and phrasing. As a result, the abstracts may not appear in search results when using these queries with traditional search engines.

The goal of this dataset is to train models to retrieve relevant documents for searches that may be overlooked by conventional term-matching approaches.

The dataset was created in August 2023 and contains 53,146 encephalitis abstracts collected using the Biopython library. GPT-3.5 was then used to generate multiple search queries for each abstract that are relevant to topics discussed in the abstract, but the wording/phrasing is different from what is contained in the abstract.

By training on this data, more semantic models could better connect user search queries with relevant content. This has the potential to improve search recall for specialized domains like medical literature.


To open
```
import pandas as pd
df = pd.read_parquet('raw_training_df.parquet')
```