Key error: 0 in DataCollatorForSeq2Seq for BERT
Hello everyone,I am trying to fine-tune a german BERT2BERT model for text summarization unsingbert-base-german-casedand want to use dynamic padding. However, when callingTrainer.train()I receive an error, that tensors cannot be created and I should use padding. I was able to trace this error back to my DataCollator. The code I used is the following:First, I define the function to tokenize my data and do so using the map function.tokenizer = BertTokenizerFast.from_pretrained(“bert-base-german-cased”)tokenizer.bos_token = tokenizer.cls_tokentokenizer.eos_token = tokenizer.sep_tokenmax_input_length = 512max_target_length = 128def prepro_bert2bert(samples):model_inputs = tokenizer(samples[“text”], max_length = max_input_length, truncation = True)with tokenizer.as_target_tokenizer():
    labels = tokenizer(samples["description"], max_length = max_target_length, truncation = True)
      
samples["input_ids"] = model_inputs.input_ids
samples["attention_mask"] = model_inputs.attention_mask
samples["decoder_input_ids"] = labels.input_ids
samples["decoder_attention_mask"] = labels.attention_mask
samples["labels"] = labels.input_ids.copy()

return samplestraindata = Dataset.from_pandas(traindata)tokenized_traindata = traindata.map(prepro_bert2bert, batched = True, remove_columns = [“text”, “description”, “_index_level_0_”])tokenized_traindata.set_format(columns = [“labels”, “input_ids”, “attention_mask”, “decoder_input_ids”, “decoder_attention_mask”])My tokenized_traindata looks like the following:Dataset({features: [‘attention_mask’, ‘decoder_attention_mask’, ‘decoder_input_ids’, ‘input_ids’, ‘labels’],num_rows: 7986})Then I instantiate my bert2bert model and my DataCollator:bert2bert = EncoderDecoderModel.from_encoder_decoder_pretrained(“bert-base-german-cased”, “bert-base-german-cased”)data_collator = DataCollatorForSeq2Seq(tokenizer, model = bert2bert, padding = “longest”)Lastly, I form batches from my training data and want to use the data_collatorsamples = tokenized_traindata[:8]batch = data_collator(samples)This returns the following error messageKeyError                                  Traceback (most recent call last)in----> 1 batch = data_collator(samples)2 {k: v.shape for k, v in batch.items()}~\miniconda3\envs\BERTnew\lib\site-packages\transformers\data\data_collator.py incall(self, features)271272     defcall(self, features):→ 273         labels = [feature[“labels”] for feature in features] if “labels” in features[0].keys() else None274         # We have to pad the labels before callingtokenizer.padas this method won’t pad them and needs them of the275         # same length to return tensors.KeyError: 0Unfortunately, I do not know where to look further for a solution. I hope someone may has a suggestion where to look or how to solve this. Thank you very much in advance!
2021-06-24T15:00:35Z
This is because thedatasetslibrary returns a slice of the dataset as a dictionary with lists for each key. The data collator however expects a list of dataset elements, so a list of dictionaries. Practically, I think you need to do:samples = [tokenized_traindata[i] for i in range(8)]
batch = data_collator(samples)
2021-06-24T15:19:38Z
Thank you for your reply. I tried your suggestion and now get the first error message again:samples = [tokenized_traindata[i] for i in range(8)]batch = data_collator(samples)ValueError                                Traceback (most recent call last)~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization_utils_base.py in convert_to_tensors(self, tensor_type, prepend_batch_axis)698                 if not is_tensor(value):→ 699                     tensor = as_tensor(value)700ValueError: expected sequence of length 40 at dim 1 (got 47)During handling of the above exception, another exception occurred:ValueError                                Traceback (most recent call last)in----> 1 batch = data_collator(samples)~\miniconda3\envs\BERTnew\lib\site-packages\transformers\data\data_collator.py incall(self, features)283                 )284→ 285         features = self.tokenizer.pad(286             features,287             padding=self.padding,~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization_utils_base.py in pad(self, encoded_inputs, padding, max_length, pad_to_multiple_of, return_attention_mask, return_tensors, verbose)2678                 batch_outputs[key].append(value)2679→ 2680         return BatchEncoding(batch_outputs, tensor_type=return_tensors)26812682     def create_token_type_ids_from_sequences(~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization_utils_base.py ininit(self, data, encoding, tensor_type, prepend_batch_axis, n_sequences)202         self._n_sequences = n_sequences203→ 204         self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)205206@property~\miniconda3\envs\BERTnew\lib\site-packages\transformers\tokenization_utils_base.py in convert_to_tensors(self, tensor_type, prepend_batch_axis)713                         “Please see if a fast version of this tokenizer is available to have this feature available.”714                     )→ 715                 raise ValueError(716                     "Unable to create tensor, you should probably activate truncation and/or padding "717                     “with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.”ValueError: Unable to create tensor, you should probably activate truncation and/or padding with ‘padding=True’ ‘truncation=True’ to have batched tensors with the same length.Do you have any suggestions as to why this happens again? Thank you very much in advance for your response and time!
2021-06-24T15:24:08Z
I think it’s compaining because thedecoder_attention_maskanddecoder_input_idsaren’t all of the same length. You should let theDataCollatorForSeq2Seqcreate them for you.
2021-06-24T15:29:10Z
Thank you again! So I should not tokenize the labels and original text at the same time in myprepro_bert2bertfunction? How would I approach that?
2021-06-24T15:30:56Z
You definitely should tokenize your labels, but remove the linessamples["decoder_input_ids"] = labels.input_ids
samples["decoder_attention_mask"] = labels.attention_maskwhich the data collator will ser properly from the labels and the models (there is a shift right to apply in some models).
2021-06-24T15:42:12Z
Thank you very much! This solved my problem.
2021-06-24T16:01:21Z
Sorry to bother you again.I now wanted to implement what I learned in my training loop, but now get a value error that I must specify either input_ids or inputs_embeds. Do I need to specify anything else, so that the decoder_input_ids and decoder_attention_mask from the collator get passed into the model? My training arguments look like this:batch_size = 2learning_rate = 2e-5epochs = 1training_args = Seq2SeqTrainingArguments(output_dir = “model_checkpoints/”,evaluation_strategy = “epoch”,learning_rate = learning_rate,per_device_train_batch_size = batch_size,per_device_eval_batch_size = batch_size,save_total_limit = 3,num_train_epochs = epochs,predict_with_generate = True,fp16 = True)trainer = Seq2SeqTrainer(bert2bert,training_args,train_dataset = tokenized_traindata,eval_dataset = tokenized_valdata,data_collator = data_collator,tokenizer = tokenizer,compute_metrics = compute_metrics)Thank you again for your time and patience!
2021-06-24T16:05:41Z
Pinging@patrickvonplatensince I’m not super familiar with this BERT2BERT archittecture.
2021-09-28T23:21:13Z
Hi@Arcticweasel,I ran into the same problem with this, requiring to specify input_ids or inputs_embeds. I thought that input_ids is already in tokenized_traindata after preprocessing. Have you found a solution?Cheers!
2024-03-13T20:42:18Z
I was having a similar issue and landed on this thread.I’m trying to understand why theDataCollatorForSeq2Seqis not padding the batch inputs similar toDataCollatorWithPadding?Do I need to specifypadding=Truein the tokenizer? (only then seems to be working)When addingpadding=Trueas arguement to theDataCollatorForSeq2Seqit doesn’t work?MWE:from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, DataCollatorForSeq2Seq, Seq2SeqTrainer
from datasets import load_dataset

model_checkpoint = "facebook/bart-base"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

english_dataset = load_dataset("McAuley-Lab/Amazon-Reviews-2023", "raw_review_All_Beauty")

def preprocess_function(examples):
    model_inputs = tokenizer(examples["text"],)
    # Set up the tokenizer for targets
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["title"])

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_datasets = english_dataset.map(preprocess_function, batched=True)

tokenized_datasets = tokenized_datasets.remove_columns(['rating', 'title', 'text', 'images', 'asin', 'parent_asin', 'user_id', 'timestamp', 'helpful_vote', 'verified_purchase'])

tokenized_datasets['full'][:2].keys()
# dict_keys(['input_ids', 'attention_mask', 'labels'])

features = [tokenized_datasets["full"][i] for i in range(2)]

data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)

featuers_padded = data_collator(features)

len(features_padded[0]['input_ids'])
# 37

len(features_padded[1]['input_ids'])
# 69

len(features_padded[0]['labels'])
# 53

len(features_padded[1]['labels'])
# 82Any idea what’s going on here?