How to use dataset with costume function?
I want to callDatasetDictmapfunction with parameters, and I dont know how to do it.I have function with the following API:def tokenize_function(tokenizer, examples):
    s1 = examples["premise"]
    s2 = examples["hypothesis"]
    args = (s1, s2)
    return tokenizer(*args, padding="max_length", truncation=True)And when I’m trying to use in this way:dataset = load_dataset("json", data_files=data_files)
   tokenizer          = AutoTokenizer.from_pretrained(model_name)
   tokenized_datasets = dataset.map(tokenize_function, tokenizer, batched=True)I’m getting error:TypeError: list indices must be integers or slices, not strHow can I callmapfunction in my example ?
2022-08-08T17:03:16Z
The function shouldn’t take the tokenizer as input, only the examples. You can add additional arguments to the function, which can then be passed as fn_kwargs in the map method.Option 1: define tokenizer outside the functionfrom transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(model_name)

dataset = load_dataset("json", data_files=data_files)

def tokenize_function(examples):
    s1 = examples["premise"]
    s2 = examples["hypothesis"]
    args = (s1, s2)
    return tokenizer(*args, padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, tokenizer, batched=True)Option 2: usefn_kwargsto pass the tokenizer to the functionfrom transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(model_name)

dataset = load_dataset("json", data_files=data_files)

def tokenize_function(examples, tokenizer):
    s1 = examples["premise"]
    s2 = examples["hypothesis"]
    args = (s1, s2)
    return tokenizer(*args, padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, fn_kwargs={"tokenizer":tokenizer}, batched=True)
2023-06-19T02:14:18Z
how can i do a function on Dataset as batch here my code:def get_sent(examples):sents =sentiment_classifier(examples[‘text’])labels=[sent[0][‘label’] for sent in sents]examples[‘label’]=labelsreturn examplesto_label1 = to_label1.map(get_sent,batched=True)when i run the code i got error and cant run function as batch.
2023-06-19T15:53:17Z
Can you paste the error message?