Load pretrained model's tokenizer with or without vocabulary?
I am having some confusion loading a pre-trained tokenizer of a model. I’m trying to use someone’s model from their repo, they have following saved:checkpoints of their fine-tuned bert uncased modelvocabulary files (multiple), probably all generated after training aBertWordPieceTokenizerNow, if I want to fine-tune model on my data, how do I load pre-trained tokenizer and use it on my dataset? Do I use vocabulary files or should I load my model as tokenizer as I have seen in an example here:pic1847×295 50.3 KBPlease guide me. Thanks
2024-08-28T09:26:47Z
hi@Sabs101Please checkFine-tuning a model with the Trainer API - Hugging Face NLP Course.The most important part should be writing tokenize_function if you need:def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)huggingface.coProcessing the data - Hugging Face NLP CourseWe’re on a journey to advance and democratize artificial intelligence through open source and open science.We can help more if you share your initial model(probably it’s bert-base-uncased) and some lines from your dataset.
2024-08-30T04:40:14Z
Hi @mahmutc,Thankyou for sharing link with me, but confusion still persists. I want to know how I can load my tokenizer (pre-trained) for using it on my own datasaet, should I load it as I load the model or if vocab file is present with the model, can I do.from_pretrained(‘vocab.txt’)and load my tokenizer?The model I am using is a fine-tuned version of original ‘bert-base-uncased’, fine-tuned for a different language. Here is what I have from the model:pic1220×1500 78.9 KBAttaching a link to it as well, if it helpsRoman_Urdu_BERT/roman_urdu at master · usamakh20/Roman_Urdu_BERT · GitHub