task_categories:feature-extraction task_categories:text-classification language:de license:cc-by-sa-4.0 size_categories:100K<n<1M format:csv modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: cc-by-sa-4.0
language:
- de
size_categories:
- 100K<n<1M
task_categories:
- feature-extraction
- text-classification
---

This is a filtered version of Philip May's German paraphrase dataset.

The dataset has been filtered for the sake of convenience, since smaller devices do not support such large files.

All text pairs in the dataset are paraphrases, and are therefore labelled 1. As such, the dataset is well-suited for use in conjunction with the multiple negatives ranking loss.

As the original author suggests, the dataset has been filtered, mostly following the guidelines set by the author. Any row that doesn't comply with the following conditions was filtered out:
- min_char_len < 25
- de_token_count > 30
- en_de_token_count > 30
- jaccard_similarity > 0.3
- cos_sim < 0.9

## Licensing

Copyright (c) 2022 [Philip May](https://may.la/), [Deutsche Telekom AG](https://www.telekom.com/).

This work is licensed under [CC-BY-SA 4.0]([https://link-url-here.org](https://creativecommons.org/licenses/by-sa/4.0/).