Training a Tokenizer on a Streamed Dataset
Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions providedhere, with the addition ofstreaming=Trueduring the dataset loading step. However, it quickly failed as theIterableDatasetclass does not have a length property (unlike the normalDatasetclass).How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s theColab linkto reproduce the error.Any help would be greatly appreciated. Thanks!Note: Thisissueseems to be related to mine.
2021-07-06T12:51:29Z
Hi !You can follow the instructionshereand use this batch iterator instead:def batch_iterator(batch_size=1000):
    batch = []
    for example in dataset:
        batch.append(example["text"])
        if len(batch) == batch_size:
            yield batch
            batch = []
    if batch:  # yield last batch
        yield batch
2021-07-06T13:04:01Z
Thanks for the reply and solution@lhoestq.Appreciate the effort you and your team are doing!Cheers!
2023-01-12T14:00:37Z
I’m having problems to train the tokenizers on a very large dataset. I think the problem is in the “yield” done in the interator. Any change I could train the tokenizer without the yield?
2023-05-30T03:12:00Z
@w11woYou can do this to get the__len__property:dataset.with_format("torch")
2024-01-31T06:52:51Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.