size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
dataset_info:
  features:
  - name: Compound Word
    dtype: string
  - name: Word1
    dtype: string
  - name: Word2
    dtype: string
  splits:
  - name: train
    num_bytes: 88123
    num_examples: 1653
  download_size: 35050
  dataset_size: 88123
---
# Compound Words Finnish Dataset

Finnish, like many Uralic languages, is characterized by its complex morphological structure, particularly the use of compound words. These compounds challenge natural language processing (NLP) systems due to their flexible formation and frequent absence in dictionaries. To address this issue, a dataset was created to decompose Finnish compound words into their component parts, supporting NLP applications such as machine translation, search engines, and linguistic research.

## Methodology
#### Data Collection
Text data was gathered from diverse sources to ensure a wide representation of Finnish compounds. News articles were dynamically extracted from Yle.fi using BeautifulSoup, public domain books were downloaded from Project Gutenberg and processed with PyPDF2, and additional literary and historical texts were retrieved from Finna.fi. The collected text was consolidated into a single file for preprocessing.

#### Preprocessing
The raw text was cleaned using regular expressions to remove punctuation, HTML tags, and extra spaces. Tokenization was applied to split the text into individual words, and words longer than ten characters were filtered as potential compound candidates. This threshold ensured focus on the complex structures typical of 
Finnish compounds.

#### Compound Word Identification
Each candidate was cross-referenced with an open-source Finnish dictionary called Omorfi. Words not present in the dictionary were flagged as compounds. These flagged words were decomposed by splitting them iteratively and checking if the resulting components existed in the dictionary. Special linguistic rules were applied to account for Finnish linking vowels, such as "s," and common suffixes like -ssa and -t.

### Validation and Data et Creation
To ensure the accuracy of the dataset, manual validation was performed on a sample of decompositions. Additionally, GPT-4 was used to identify false positives or incorrect splits, which were subsequently removed. The final dataset was stored in CSV format, containing each compound word along with its decomposed components.

### Applications and Contributions
This dataset has several practical applications:

- **Machine Translation**: Enables better handling of Finnish compound words by decomposing them into their base components for accurate translation.
- **Search Engines**: Improves query matching by breaking down compound search terms into their parts.
- **Language Learning**: Assists learners in understanding and interpreting complex Finnish words.
- **Linguistics Research**: Provides insights into the formation and structure of Finnish compounds, supporting studies in Uralic morphology.

Unlike existing resources such as Omorfi, which provide general morphological analysis, this dataset specifically focuses on decomposing compound words, addressing a key gap in Finnish NLP resources.

## Dataset Format

The dataset includes the following columns:
- `compound_word`: The original Finnish compound word.
- `word1`: The first part of the compound word.
- `word2`: The second part of the compound word.

## Example Entries

| Compound Word          | Word1         | Word2         |
|------------------------|---------------|---------------|
| kaupunginsuunnittelu   | kaupunki      | suunnittelu   |
| terveysasemat          | terveys       | asemat        |

### References
1.	Yleisradio Oy (Yle). https://yle.fi.
2.	Omorfi: Open Morphology for Finnish. https://github.com/flammie/omorfi.
3.	Project Gutenberg: Public Domain Texts. https://www.gutenberg.org.
4.	Finna.fi: The National Digital Library of Finland. https://finna.fi.

### Citation
Please cite the data with the following citation:

Melany Macias. *Breaking Down Finnish Compounds: Creating a Dataset for NLP*. Lightning Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages, Nov 2024, Helsinki, Finland. [hal-04801912](https://hal.archives-ouvertes.fr/hal-04801912).

Hämäläinen, M., & Pirinen, F. (2024, noviembre 28). Lightning Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages. Lightning Proceedings of the 9th International Workshop on Computational Linguistics for Uralic Languages. The 9th International Workshop on Computational Linguistics for Uralic Languages (IWCLUL), Helsinki, Finland. https://doi.org/10.5281/zenodo.14212595