Dataset map method - how to pass argument to the function
Hi, just started using the Huggingface library. I am wondering how can I passmodelandtokenizerto my processing function along with the batch when using themapmethod.def my_processing_func(batch, model, tokenizer):–code–I am using map like this…new_dataset = my_dataset.map(my_processing_func, model, tokenizer, batched=True)when I do this it does not fail but instead of passing the dictionary with input_ids and attention_mask, it passes alistof just input_ids as the batch to my_processing_func. When I remove the model and tokenizer argument then it sends the dictionary as expected.Where am I going wrong?Thanks in advance.
2022-03-31T11:24:25Z
Hi! You can usefn_kwargsto pass the arguments to the map function:new_dataset = my_dataset.map(my_processing_func, batched=True, fn_kwargs={"model": model, "tokenizer": tokenizer})Or you can usepartial:from functools import partial
new_dataset = my_dataset.map(partial(my_processing_func, model=model, tokenizer=tokenizer), batched=True)
2022-03-31T11:51:59Z
Is there any downside to using either options? If I remember correctly (?) lambdas are not picklable. So my assumption would be that if you do something likenew_dataset = my_dataset.map(lambda batch: my_processing_func(batch, model, tokenizer), batched=True)it won’t be cached. Is that correct?
2022-03-31T13:05:33Z
Super!! this works for me … thanks
2022-04-05T12:16:51Z
There shouldn’t be a significant difference in speed between these two approaches.We usedill, which knows how to pickle lamdas in most situations.