Building Transformer Tokenizers (Dhivehi NLP #1)
🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Get in touch with Ashraq:
https://www.linkedin.com/in/ismailashraq/

The language of Dhivehi (or Maldivian) is fascinating. It uses a complex writing system known as Thaana, and I absolutely cannot comprehend any of it. It is so wildly different from anything I know - but, like the archipelago, it looks wonderful.

Ashraq described the difficulty of applying NLP to his native tongue of Dhivehi. There are several reasons for this, which we will explore in this video, and learn how to build an effective Dhivehi WordPiece tokenizer.

📕 Article:
https://towardsdatascience.com/designing-tokenizers-for-low-resource-languages-7faa4ab30ef4

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Article Friend Link (Free Access):
https://towardsdatascience.com/designing-tokenizers-for-low-resource-languages-7faa4ab30ef4?sk=c0c16de9eea7dbe1d2a9c106abf38e1a

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:06 Dhivehi Project
02:28 Hurdles for Low Resource Domains
04:21 Dhivehi Dataset
04:52 Download Dhivehi Corpus
08:25 Tokenizer Components
08:44 Normalizer Component
11:55 Pre-tokenization Component
14:59 Post-tokenization Component
16:26 Decoder Component
17:41 Tokenizer Implementation
21:04 Tokenizer Training
24:22 Post-processing Implementation
27:12 Decoder Implementation
28:07 Saving for Transformers
30:33 Tokenizer Test and Usage
31:36 Download Dhivehi Models
32:21 First Steps