How to use models in tranformers?
I am very new to using transformers. I have just started using them, and I require them for a classification task, Where I have tweets from topics, and I need to find their hate score. I have been suggested to use theGroNLP/hateBERT · Hugging Facemodel to do so.But, I was trying it out by the following code:from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")

model = AutoModelForMaskedLM.from_pretrained("GroNLP/hateBERT")Then I used the following code:from transformers import pipeline

classifier = pipeline('text-classification', model=model, tokenizer=tokenizer)But I am getting an error:The model 'TFBertLMHeadModel' is not supported for text-classification. Supported models are ['TFAlbertForSequenceClassification', 'TFBertForSequenceClassification', 'TFCamembertForSequenceClassification', 'TFConvBertForSequenceClassification', 'TFCTRLForSequenceClassification', 'TFDebertaForSequenceClassification', 'TFDebertaV2ForSequenceClassification', 'TFDistilBertForSequenceClassification', 'TFElectraForSequenceClassification', 'TFFlaubertForSequenceClassification', 'TFFunnelForSequenceClassification', 'TFGPT2ForSequenceClassification', 'TFGPTJForSequenceClassification', 'TFLayoutLMForSequenceClassification', 'TFLayoutLMv3ForSequenceClassification', 'TFLongformerForSequenceClassification', 'TFMobileBertForSequenceClassification', 'TFMPNetForSequenceClassification', 'TFOpenAIGPTForSequenceClassification', 'TFRemBertForSequenceClassification', 'TFRobertaForSequenceClassification', 'TFRoFormerForSequenceClassification', 'TFTapasForSequenceClassification', 'TFTransfoXLForSequenceClassification', 'TFXLMForSequenceClassification', 'TFXLMRobertaForSequenceClassification', 'TFXLNetForSequenceClassification'].Can someone suggest how to modify this code so I can use the hateBERT model to find the hateful score? And any resources to learn about using any generic models would be very helpful.
2022-10-03T16:34:29Z
Hello and welcome@selfisheagle!You’re getting an error because you loaded a model withAutoModelForMaskedLMinstead ofAutoModelForSequenceClassification, which is what you want for your classification task:from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")
model = AutoModelForSequenceClassification.from_pretrained("GroNLP/hateBERT")
classifier = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)For more resources, check out thedocsabout how to useAutoModelto load a model for a task orChapter 2of the Hugging Face course.
2022-10-03T16:55:17Z
Thanks for your response.When I ran the code you gave by replacing the sentiment-analysis part with text classification, I thought it would give hate vs no hate classification. But instead, I get the response.Some weights of the model checkpoint at GroNLP/hateBERT were not used when initializing BertForSequenceClassification: [‘cls.predictions.decoder.weight’, ‘cls.predictions.transform.LayerNorm.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.transform.dense.bias’, ‘cls.predictions.decoder.bias’, ‘cls.predictions.transform.dense.weight’, ‘cls.predictions.bias’]This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).Some weights of BertForSequenceClassification were not initialized from the model checkpoint at GroNLP/hateBERT and are newly initialized: [‘classifier.weight’, ‘classifier.bias’]You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.So, do I have to train still the model for hate vs no hate classification, I thought hateBERT would already have done something along those lines, and I don’t have to pre-train the model.