task_categories:text-generation language:no language:nn language:nb license:cc-by-4.0 size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2412.09460 region:us
---
dataset_info:
- config_name: nb
  features:
  - name: idiom
    dtype: string
  - name: completion
    dtype: string
  splits:
  - name: test
    num_bytes: 13208
    num_examples: 402
  download_size: 10855
  dataset_size: 13208
- config_name: nn
  features:
  - name: idiom
    dtype: string
  - name: completion
    dtype: string
  splits:
  - name: test
    num_bytes: 17634
    num_examples: 401
  download_size: 13104
  dataset_size: 17634
configs:
- config_name: nb
  data_files:
  - split: test
    path: nb/test-*
- config_name: nn
  data_files:
  - split: test
    path: nn/test-*
license: cc-by-4.0
task_categories:
- text-generation
language:
- 'no'
- nn
- nb
---

 # NorIdiom 

This dataset was created for the Mímir project, as described in "The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective" [link to preprint](https://arxiv.org/abs/2412.09460)  
The idioms in this dataset are manually curated, and some idioms may be less frequent/representative of Norwegian language than others.  
For a frequency curated Norwegian idiom dataset, see [Sprakbanken/Norwegian_idioms](https://huggingface.co/datasets/Sprakbanken/Norwegian_idioms).

This dataset contains 803 Norwegian idioms, 401 of which are Norwegian nynorsk and 402 that are Norwegian bokmål.

## Usage
This dataset can be used to evaluate a language model's abilty to complete Norwegian idioms.  
The `idiom` feature contains the n-1 first words of an idiom, and the `completion` feature is the last word that completes the idiom in each row.  

Load the Norwegian bokmål idioms like this:
```python
from datasets import load_dataset

ds = load_dataset("mimir-project/noridiom", split="test", name="nb")
```
and the Norwegian nynorsk idioms like this:
```python
from datasets import load_dataset

ds = load_dataset("mimir-project/noridiom", split="test", name="nn")
```



 