Training and Testing an Italian BERT - Transformers From Scratch #4
We need two things for training, our DataLoader and a model. The DataLoader we have — but no model.

For training, we need a raw (not pre-trained) RobertaForMaskedLM. To create that, we first need to create a RoBERTa config object to describe the parameters we’d like to initialize FiliBERTo with.

Once we have our model, we set up our training loop and train!

Post-training, we'll test the model with Laura, who is Italian - and hope for the best.

Part 1: https://youtu.be/GhGUZrcB-WM
Part 2: https://youtu.be/JIeAB8vvBQo
Part 3: https://youtu.be/heTYbpr9mD8
---

📙 Medium article:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6?sk=9db6224efbd4ec6fd407a80b528e69b0

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:35 Review of Code
02:02 Config Object
06:28 Setup For Training
10:30 Training Loop
14:57 Dealing With CUDA Errors
16:17 Training Results
19:52 Loss
21:18 Fill-mask Pipeline For Testing
21:54 Testing With Laura