How to train huggingface model with fp16?
Hi I am using pytorch and huggingface to train my roberta-base to RTE dataset.But I find that just usingtorch.halfto my model will cause nan after first backward.Is there any way to train my model with fp16 and without using huggingface’sTrainerfunction?
2022-05-23T23:08:54Z
Yes, you can do this without using the Huggingface trainer. You’ll have to directly use the PyTorch AMP feature though (and this is what the HF trainer does as well, if you specify “amp” for thehalf_precision_backend). You’d want to get started by readingthispage.