Training BERT #2 - Train With Masked-Language Modeling (MLM)
🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

BERT has enjoyed unparalleled success in NLP thanks to two unique training approaches, masked-language modeling (MLM), and next sentence prediction (NSP).

In many cases, we might be able to take the pre-trained BERT model out-of-the-box and apply it successfully to our own language tasks.

But often, we might need to pre-train the model for a specific use case even further.

Further training with MLM allows us to tune BERT to better understand the particular use of language in a more specific domain.

Out-of-the-box BERT - great for general purpose use. Fine-tuned with MLM BERT - great for domain-specific use.

In this video, we'll cover exactly how to fine-tune BERT models using MLM in PyTorch.

👾 Code:
https://github.com/jamescalam/transformers/blob/main/course/training/03_mlm_training.ipynb

Meditations data:
https://github.com/jamescalam/transformers/blob/main/data/text/meditations/clean.txt

Understanding MLM:
https://youtu.be/q9NS5WpfkrU

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c?sk=17a19eca8dc8280bea4138802580ffe0

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff