aloobun/IN-Llama-3-Tokenizer

In this experiment, I trained a tokenizer that supports multiple Indian languages and merged it with the Llama-3 tokenizer.

STEP 1:

I sampled data from the multilingual(7 Indic languages) aloobun/dhpileIN dataset and trained a SentencePiece tokenizer.

STEP 2:

I evaluated the tokenizer's performance on:

Unicode coverage.
Token distribution.
Tokenization complexity across different scripts.
Encoding and decoding capabilities &
Edge cases e.g., special characters, numbers, etc.

STEP 2.1:

The first test gives detailed results of the tokenizer's performance on unicode coverage, token distribution visualiztion and complexity across scripts.

Step 2.2:

The second script tests the encoding and decoding capabilities. Example output:

Bengali Analysis:
Original Text Length: 48 characters
Token IDs Count: 11
Token Strings: ['▁আমি', '▁বাংলাদেশ', '▁থেকে', '▁এসে', 'ছি', '।', '▁কলকাতা', '▁একটি', '▁সুন্দর', '▁শহর', '।']
Text Reconstruction: True

Hindi Analysis:
Original Text Length: 49 characters
Token IDs Count: 15
Token Strings: ['▁नम', 'स्ते', ',', '▁मैं', '▁भारत', '▁से', '▁हू', 'ँ', '।', '▁दिल्ली', '▁बहुत', '▁बड़ा', '▁शहर', '▁है', '।']
Text Reconstruction: True

Kannada Analysis:
Original Text Length: 53 characters
Token IDs Count: 13
Token Strings: ['▁ನಾನು', '▁ಬೆಂಗಳೂರಿ', 'ನಿಂದ', '▁ಬಂದ', 'ಿದ್ದೇನೆ', '।', '▁ಕನ್ನಡ', '▁ಒಂದು', '▁ಸೋ', 'ಂಪ', 'ಿನ', '▁ಭಾಷೆ', '।']
Text Reconstruction: True

Malayalam Analysis:
Original Text Length: 47 characters
Token IDs Count: 15
Token Strings: ['▁ഞ', 'ാ', 'ൻ', '▁കേരള', 'ത്തി', 'ൽ', '▁നിന്നാണ്', '.', '▁കൊച്ചി', '▁ഒരു', '▁സുന്ദ', 'ര', '▁നഗ', 'രം', '.']
Text Reconstruction: True

Telugu Analysis:
Original Text Length: 53 characters
Token IDs Count: 10
Token Strings: ['▁నేను', '▁తెలంగాణ', '▁నుంచి', '▁వచ్చ', 'ాను', '.', '▁హైదరాబాద్', '▁అద్భుతమైన', '▁నగరం', '.']
Text Reconstruction: True

Tamil Analysis:
Original Text Length: 54 characters
Token IDs Count: 13
Token Strings: ['▁நான்', '▁தமிழ்நா', 'ட்டை', 'ச்', '▁சேர்ந்த', 'வன்', '.', '▁சென்னை', '▁ஒரு', '▁பெரிய', '▁நக', 'ரம்', '.']
Text Reconstruction: True

Gujarati Analysis:
Original Text Length: 50 characters
Token IDs Count: 12
Token Strings: ['▁હું', '▁ગુજરાત', '▁થી', '▁આવ્યો', '▁છું', '।', '▁અમદાવાદ', '▁એક', '▁સુંદર', '▁શહેર', '▁છે', '।']
Text Reconstruction: True

STEP 3:

This script is used to merge and extend the tokenizer for the Llama3 tokenizer.

Script ensures:

No duplicate tokens are added.
Tokens arent excessively long.
New tokens are correctly integrated.
Token mappings, etc

I feel there are some unecessary bloat like token validation and redundant test methods in the script. I'm still working on how to improve things and will update as soon as I have any progress.

Here's a comparison of sub word fertility scores between sarvam-1 and this model.

	sarvam-1	IN-Llama-3-Tokenizer
Bengali	1.7	3.52
Gujrati	2.784313	3.588235
Hindi	1.583333	2.933333
Kannada	2.571428	3.976190
Malayalam	3.487804	4.365853
Tamil	2.767441	3.860465
Telugu	2.372093	3.511627