ruthuvikas1998/kannada-tokenizer

Kannada ByteLevel BPE Tokenizer

This repository contains a Byte-Level BPE (Byte Pair Encoding) tokenizer for the Kannada language, designed using the Hugging Face tokenizers library. The tokenizer is optimized for handling Kannada text, including pure Kannada, mixed-language text, numbers, punctuation, and special cases like URLs and emojis.

Features

• ByteLevel BPE Tokenization: Ensures efficient and compact subword representation.
• Special Tokens Support: Includes special tokens for sentence boundaries, padding, and masking.
• Post-processing similar to BERT: Helps structure input data effectively for downstream models.
• Handles Kannada and mixed-language text: Can tokenize Kannada-English text seamlessly.

Usage

This tokenizer is pre-trained on a diverse Kannada text corpus and can be used for a variety of NLP tasks such as text classification, language modeling, and machine translation.

Training Details

The tokenizer was trained on a large Kannada corpus, including Wikipedia articles, literary texts, and online content, ensuring a broad vocabulary coverage.

Supported Test Cases

The tokenizer has been tested on multiple text categories:

• Pure Kannada: Simple and complex Kannada sentences.
• Mixed Language: Kannada-English hybrid text.
• Numbers and Dates: Kannada and Western numerals, date formats, and currency symbols.
• Punctuation Handling: Sentences with punctuation and special characters.
• Special Cases: URLs, hashtags, emojis, and file paths.

Tokenizer Test Results

Category: Pure Kannada

Test Case 1: Basic sentence

Original text: ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ
Encoded tokens: ['<s>', 'ನಮಸ', 'à³į', 'à²ķ', 'ಾ', 'ರ', 'Ġà²ķನ', 'à³į', 'ನಡ', 'Ġà²Ń', 'ಾ', 'ಷ', 'à³Ĩ', '</s>']
Token IDs: [0, 1461, 264, 278, 270, 272, 738, 264, 407, 386, 270, 323, 268, 1]
Decoded text: ನಮಸ್ಕಾರ ಕನ್ನಡ ಭಾಷೆ
Analysis:

• Number of tokens: 14
• Average token length: 1.29 characters
• Reconstruction: Perfect

Test Case 2: Complex sentence

Original text: ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ
Encoded tokens: ['<s>', 'à²ķನ', 'à³į', 'ನಡ', 'Ġನ', 'ಾ', 'ಡ', 'ಿ', 'ನ', 'Ġಸ', 'à²Ĥ', 'ಸ', 'à³į', 'à²ķ', 'à³ĥ', 'ತ', 'ಿ', 'Ġಮತ', 'à³į', 'ತ', 'à³ģ', 'Ġಪರ', 'à²Ĥ', 'ಪರ', 'à³Ĩ', '</s>']
Token IDs: [0, 754, 264, 407, 298, 270, 280, 267, 266, 300, 275, 281, 264, 278, 412, 271, 267, 382, 264, 271, 265, 360, 275, 524, 268, 1]
Decoded text: ಕನ್ನಡ ನಾಡಿನ ಸಂಸ್ಕೃತಿ ಮತ್ತು ಪರಂಪರೆ
Analysis:

• Number of tokens: 26
• Average token length: 1.27 characters
• Reconstruction: Perfect

Category: Mixed Language

Test Case 1: Kannada with English

Original text: ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ
Encoded tokens: ['<s>', 'ನನ', 'à³į', 'ನ', 'Ġ', 'e', 'm', 'a', 'i', 'l', 'Ġ', 'I', 'D', 'Ġà²ĩದ', 'à³ģ', 'Ġ', 'e', 'x', 'a', 'm', 'p', 'l', 'e', '@', 'e', 'm', 'a', 'i', 'l', '.', 'com', 'Ġà²Ĩà²Ĺ', 'ಿ', 'ದ', 'à³Ĩ', '</s>']
Token IDs: [0, 306, 264, 266, 225, 73, 81, 69, 77, 80, 225, 45, 40, 493, 265, 225, 73, 92, 69, 81, 84, 80, 73, 36, 73, 81, 69, 77, 80, 18, 469, 408, 267, 269, 268, 1]
Decoded text: ನನ್ನ email ID ಇದು [email protected] ಆಗಿದೆ
Analysis:

• Number of tokens: 36
• Average token length: 1.14 characters
• Reconstruction: Perfect

Repository Structure

The repository consists of tokenizer files, configuration files, and documentation:

├── kannada_tokenizer/
│   ├── vocab.json
│   ├── merges.txt
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── special_tokens_map.json
│   ├── config.json
├── README.md

License

This tokenizer is released under the MIT License.

Citation

If you use this tokenizer, please cite this repository.

For improvements and contributions, feel free to submit a Pull Request or open an Issue. 🚀