Problems and solution on Trainer
I am using the trainer to train an ASR model, the dataset and the output dimension are huge. This will cause some problems during training. I struggle with it many days, so I post my solution here, hope it can help.compute_metrics out of memory issueduring compute_metrics, it will save all the logits in an array, when the output dimension is large, it will easily cause out-of-memory on a large dataset. The solution is to use torch.argmax on logits first to avoid saving all the data.when using trainer on seq2seq model, if the model output contains past_key_value, it will cause length error when merging different output, so past_key_value needs to be dropped on model output.group_by_length will take a very long processing time on model.train, and it uses a lot of memory to calculate the length of the data.
2021-11-09T13:02:45Z
Note that for 3, you can have it in yourdatasets.Datasetcomputed once and for all with themapmethod, you just have to store the resuls in a"lengths"column. It will then uses the Dataset features and not try to access every element.
2021-11-10T12:01:31Z
model.forward should havelabelsargument if you have loss return, otherwise during prediction_loop will cause length error.it needs to add input_ids in dataset(for wav2vecProcessor, input will be input_values, I have to add an extra input_ids field XD)
2021-12-17T18:04:46Z
sgugger:lengthsIt should be length instead of lengths, it can be customize using length_column_name now! That’s nice!