DataCollatorWithPaddings without Tokenizer
I want to fine-tune a model…model = BertForTokenClassification.from_pretrained('monilouise/ner_pt_br'with this dataset:raw_datasets = load_dataset('lener_br')The raw_datasetsloaded are already tokenized and encoded. And I don’t know how it was tokenized. Now, I want to pad the inputs, but I don’t know how to useDataCollatorWithPaddingsin this case.I noticed that this dataset is similar townutdataset from the docs. Still, I can’t figure out what should I do.
2021-10-25T20:41:41Z
You can use the base BERT tokenizer I would say (since it’s a BERT model). Just make sure the pad token is compatible with what the model expects.
2021-10-25T20:44:43Z
Is there a way to check this from the downloaded model?  Or this is something I will find in the model card?
2021-10-25T20:47:43Z
Check the model configpad_token_idfield.