task_categories:text-generation license:mit region:us
---
license: mit
task_categories:
- text-generation
---
Original dataset size: 21725
Number of duplicate clusters: 2951
Files in duplicate cluster: 8054
Unique files in duplicate cluster: 3781
Filtered dataset size: 17452
Time to deduplicate dataset: 7.37
Size of deduplicated dataset: 17452, old dataset size 21725