license:cc0-1.0 size_categories:1K<n<10K format:csv modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc0-1.0
---

# NCBI Disease Corpus for Binary Sequence Classification

## Description

This dataset is part of the MSc dissertation study titled 'Investigating the Potential of Identifying Kidney Disease-Related Articles Using Transformer Models and Large Language Models' at the University of Southampton. It is a modified version of the [NCBI Disease Corpus](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3951655/), with a binary label added to each sample. The binary label indicates whether the sample contains disease concepts (Class 1) or not (Class 0).

## Dataset Structure

The dataset is split into train and test sets:

<table> <tr> <td></td> <td><strong>Class 1</strong></td> <td><strong>Class 0</strong></td> <td><strong>Total Samples per Split</strong></td> </tr> <tr> <td><strong>Train</strong></td> <td>3,419</td> <td>2,938</td> <td>6,357</td> </tr> <tr> <td><strong>Test</strong></td> <td>539</td> <td>402</td> <td>941</td> </tr> </table>

Columns:

- **id**: Unique identifier for each sample. The ID indicate the original index of the sample in the NCBI Disease Corpus. For example, 'test-0' indicates the first sample in the test set.
- **tokens**: The text content of the sample split into tokens.
- **ner_tags**: The named entity recognition (NER) tags for each token. The tags are 0, 1, and 2. 0 indicates that the token is not part of a disease concept, 1 indicates the beginning of a disease concept, and 2 indicates the continuation of a disease concept.
- **Text**: The joined text content of the sample.
- **labels**: The binary label for the sample. 1 indicates that the sample contains disease concepts, and 0 indicates that the sample does not contain disease concepts.
