Sequence Classification -- Fine Tune?
Hi,I am new to Transformers/NLP.I am trying to use Transformers for text classification.  If I am not classifying to one of the pre-made GLUE benchmarks (and using my own use-case classes & texts), do I have to “fine-tune” the model?  If I have 35k texts, and 2 labels (imbalanced – 98% vs 2%) can I just use the AutoModelforSequenceClassification to throw a softmax on the end of the transformer, and train that softmax?  Or do I fine-tune the whole thing, usingthis tutorial?Thanks!  I am excited about better understanding the field and more effectively using the library.
2021-01-28T01:51:11Z
Hi@AlanFeder,I’m not familiar with imbalanced datasets, but if I were you, I would try usingexamples/text-classification/run_glue.py.In this example, instead of assigning GLUE benchmarks task names, we can use our own train_file, validation_file, and test_file.python run_glue.py \
  --model_name_or_path bert-base-cased \
  --train_file train_file_name \
  --validation_file validation_file_name \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /path/to/output/You can use CSV or JSON.datasets.load_datasetis used in the script, sothe document ofdatasets.load_datasetfor local filesmay help you.I remember that columns of the following format were easier to handle, but even if the names or order of the columns are different, it may work if the code is rewritten appropriately.(Please seetransformers/run_glue.py at 5ed5a54684ef059fa4c9710858b8e03c61295914 · huggingface/transformers · GitHubfor the detail.)sentence1, sentence2, labelorsentence, labelAgain, I’m not familiar with imbalanced datasets, so I don’t think I’ve answered your question “do I have to fine-tune?” Sorry.I am just one of the users who is learning about this library, so I hope you can get more useful advice from someone who knows more than I do.
2021-01-28T07:39:00Z
Hi@AlanFeder, in addition to@yusukemori’s useful advice you might find it instructive to start by working through the following tutorial on text classification:https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynbThe short answer to your question is that you generally do have to fine-tune one of the pretrained language models likedistilbert-base-uncasedusingAutoModelForSequenceClassification.An alternative, but less performant, approach is to just use the last hidden states of the model as input features to a classifier like logistic regression. Jay Alammar has a nice example here (along with tons of great explanations about Transformers):A Visual Guide to Using BERT for the First Time – Jay Alammar – Visualizing machine learning one concept at a time.To tackle the imbalance, you could try upsampling (downsampling) the minority (majority) class or failing that weight the classes directly in the loss function of theTrainer:Trainer — transformers 4.2.0 documentationHTH!
2021-01-31T15:06:27Z
Thanks@lewtunand@yusukemorifor your help!I tried the method you mentioned in Jay Alammar’s post – it indeed worked, but had weak performance (was beaten by my “benchmark” of tfidf/logistic regression) - so I will indeed attempt to use the fine-tune withTrainer. I may try to downsample the majority class, at least at first, to make it run faster.Thanks again!