task_categories:text-generation license:mit size_categories:10K<n<100K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
license: mit
task_categories:
- text-generation
---
Original dataset size: 21725
Number of duplicate clusters: 2951
Files in duplicate cluster: 8054
Unique files in duplicate cluster: 3781
Filtered dataset size: 17452
Time to deduplicate dataset: 7.37
Size of deduplicated dataset: 17452, old dataset size 21725