Saving tokenizer's configuration
Hi,I tried to fine-tuned language model using the following codegithub.comhuggingface/notebooks/blob/master/examples/language_modeling.ipynb{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X4cRE8IbIrIV"
   },
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "MOsHUjgdIrIW",This file has been truncated.show originalThe issue I face is that the model is saved using (trainer.save_mode()), but I don’t know how save tokenizer’s results as well?I’ve tried to add tokenizer=tokenizer to trainer, but it ariesed error.any help is really appreciated.
2022-02-24T16:22:57Z
hi,As the tokenizer is AutoTokenizer, you can use tokenizer.save_pretrained(your path) to save it.