Unable to load long-t5-local-base tokenizer after adding vocabulary
I am trying to train google/long-t5-local-base to generate some demo data for me. I wrote a function that tokenized training data and added the tokens to a tokenizer. I tried to use it in a training loop, and it complained that no config.json file existed. I then tried bringing that over from the HuggingFace repo and nothing changed.How can I get the tokenizer to load properly? Originally, I had the following filesadded_tokens.jsonspecial_tokens_map.jsontokenizer.jsontokenizer_config.json
2024-04-14T14:43:21Z
Resolved                 -
2024-04-15T02:43:42Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.