How can I drop duplicates on datasets module?
I have a very large dataset (about 200 GB) which a need to certify that are not duplicates, dropping them. How can I achieve that on datasets? Thanks!
2022-07-04T14:13:02Z
One possible solution -which doesn’t exactly answer the question- is to:Convert the Dataset object to a Pandas DataFrame object (e.g.pandas.DataFrame(your_dataset)).Employpandas.DataFrame.drop_duplicatesfunction.Finally, return to a Dataset object usingdatasets.Dataset.from_pandasfunction.I had the same question with you and I couldn’t find anything better.
2022-07-04T15:08:39Z
Hi,Papers such as OPT and PaLM detail dataset deduplication with MinHash and locality-sensitive hashing (LSH) with a Jaccard similarity of .9 or so. You would have to implement these algorithms or you can use Google’s tool for deduplicating text:GitHub - google-research/deduplicate-text-datasets.HuggingFace also has an example of text deduplication in this repository here:transformers/preprocessing.py at main · huggingface/transformers · GitHubandtransformers/minhash_deduplication.py at main · huggingface/transformers · GitHub.Best,Enrico
2022-07-05T01:32:10Z
Hi,I made some adaptations to the deduplication scripts in the Code Parrot research repository. Here is what a barebones deduplication script would look like with MinHash and LSH:GitHub - conceptofmind/Huggingface-deduplicateI also opened up a new post regarding a general use case here:Minhash DeduplicationBest,Enrico