task_categories:text-classification task_categories:multiple-choice language:en language:ja license:other size_categories:n<1K region:us code annotations
---
license: other
task_categories:
- text-classification
- multiple-choice
language:
- en
- ja
tags:
- code
- annotations
size_categories:
- n<1K
---


# Dataset overview

This is a dataset for Japanese natural language processing with multi-label annotations of research field labels in the NLP domain.


Problem Setting:
- Training Data: GitHub repositories from before 2022
- Test Data: GitHub repositories from 2023
- Objective: To predict multi-labels of research field labels in the NLP domain

Data Collection:
- We used the positive examples from [awesome-japanese-nlp-classification-dataset](https://huggingface.co/datasets/taishi-i/awesome-japanese-nlp-classification-dataset).

Dataset Features:
- A multimodal dataset including GitHub summaries, README files, PDF files, and screenshot images.
- The annotation labels were assigned with reference to [Exploring-NLP-Research](https://github.com/sebischair/Exploring-NLP-Research).

If you need image data and PDF files, please submit a request to the community.
This dataset uses text extracted from PDF files and does not include image data.

Based on [GitHub's terms of service](https://docs.github.com/ja/site-policy/acceptable-use-policies/github-acceptable-use-policies), please use this dataset for research purposes only.


# How to use this dataset

How to load in Python.

```python
from datasets import load_dataset

dataset = load_dataset("taishi-i/awesome-japanese-nlp-multilabel-dataset")
```

Details of the dataset.

```python
DatasetDict({
    train: Dataset({
        features: ['is_exist', 'url', 'created_at', 'description', 'pdf_text', 'readme_text', 'nlp_taxonomy_classifier_labels', 'awesome_japanese_nlp_labels'],
        num_rows: 407
    })
    validation: Dataset({
        features: ['is_exist', 'url', 'created_at', 'description', 'pdf_text', 'readme_text', 'nlp_taxonomy_classifier_labels', 'awesome_japanese_nlp_labels'],
        num_rows: 17
    })
    test: Dataset({
        features: ['is_exist', 'url', 'created_at', 'description', 'pdf_text', 'readme_text', 'nlp_taxonomy_classifier_labels', 'awesome_japanese_nlp_labels'],
        num_rows: 60
    })
})
```

Here is a sample of the dataset.


```json
{
  "is_exist": True,
  "url": "https://github.com/scriptin/jmdict-simplified",
  "created_at": "2016-02-07T16:34:32Z",
  "description": "JMdict and JMnedict in JSON format",
  "pdf_text": "scriptin / jmdict-simplified\nPublic\nBranches\nTags\n",
  "readme_text": "# jmdict-simplified\n\n**[JMdict][], [JMnedict][], [Kanjidic][], and [Kradfile/Radkfile][Kradfile] in JSON format**<br>\n",
  "nlp_taxonomy_classifier_labels": [
    "Multilinguality"
  ],
  "awesome_japanese_nlp_labels": [
    "Annotation and Dataset Development",
    "Vocabulary, Dictionary, and Language Input Method"
  ]
}
```

# Baseline

The baseline model was used for [TimSchopf/nlp_taxonomy_classifier](https://huggingface.co/TimSchopf/nlp_taxonomy_classifier).

The fine-tuned model was trained using this dataset to fine-tune the baseline model.

| Classification Method | Model                                                                                         | Description | Dev Prec. | Dev Rec. | Dev F1 | Eval Prec. | Eval Rec. | Eval F1 |
|-----------------------|-----------------------------------------------------------------------------------------------|-------------|-----------|----------|--------|------------|-----------|---------|
| Random Prediction     | -                                                                                             | -           | 0.034     | 0.455    | 0.064  | 0.042      | 0.513     | 0.078   |
| Baseline              | [TimSchopf/nlp_taxonomy_classifier](https://huggingface.co/TimSchopf/nlp_taxonomy_classifier) | ✓           | 0.538     | 0.382    | 0.447  | 0.360      | 0.354     | 0.349   |
| Fine-Tuning           | [TimSchopf/nlp_taxonomy_classifier](https://huggingface.co/TimSchopf/nlp_taxonomy_classifier) | ✓           | 0.538     | 0.509    | 0.523  | 0.436      | 0.484     | 0.521   |
| Zero-Shot             | gpt-4o-2024-08-06                                                                             | ✓           | 0.560     | 0.255    | 0.350  | 0.476      | 0.184     | 0.265   |


# License

We collect and publish this dataset under [GitHub Acceptable Use Policies - 7. Information Usage Restrictions](https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies#7-information-usage-restrictions) and [GitHub Terms of Service - H. API Terms](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#h-api-terms) for research purposes. This dataset should be used solely for research verification purposes. Adhering to GitHub's regulations is mandatory.
