Finetuning a specific task when pretrained model isn't trained on that specific task? Using the task model vs using the base model
I want to fine tune aRobertaForSequenceClassificationtask onmicrosoft/codebert-basemodel. Thismicrosoft/codebert-basemodel hasn’t been trained for  Sequence-Classification task.Can I load this pre-trained model inside  a SequenceClassification function and fine tune it on my dataset?model = RobertaForSequenceClassification.from_pretrained( "microsoft/codebert-base" )

Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.dense.weight', 'classifier.out_proj.weight', 'classifier.out_proj.bias', 'classifier.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.While loading I get this message which is expected as the model isn’t trained on the task and thus would not have the weights.Can I proceed with fine-tuning thisRobertaForSequenceClassificationmodel  or would I need to define my own classifier layer on top ofRobertaModeland train that?
2020-09-14T06:51:35Z
Hi@mayanksatnalikaYes, you can load a pre-trained base model for SequenceClassification.RobertaForSequenceClassificationadds the classification head itself, you won’t need to do that manually. So you can fine-tune it for classification.
2020-09-14T06:57:21Z
Thank you@valhallafor the quick reply.Just trying to understand it, I came across thishttps://github.com/curiousily/Getting-Things-Done-with-Pytorch/blob/master/08.sentiment-analysis-with-bert.ipynbhere a classifier is added on top of base model and trained rather than directly using a …forSequenceClassification task,  would it be possible to tell me how the 2 approaches differ or any relevant links on it
2020-09-14T07:01:27Z
ForSequenceClassificationmodels does pretty much the same. It takes a base model and adds pooler and classfier head on top of it, so you won’t have to do it manually.  You can see how it’s donehere, its’ pretty easy to follow
2020-09-14T07:11:59Z
Thanks a lot