Zero-shot classification fine-tuning
Hello! I’m trying to figure out how to fine-tune bert-base-uncased to perform zero-shot classification. I managed to get it working but I’m not sure I’m preparing the data in the right way:I initialize my model with theproblem_type="multi_label_classification"setting so it uses a sigmoid loss function as opposed to softmax.Then I prepare my data in the following wayI tokenize the input string and the label together usingtokenizer(sentence,label, truncation=True)and save it in theinput_idsfield of my dataset.Then I also convert the label as aone hot vectorand keep it in thelabelsfield of my dataset.Finally I ran the training with about 10k lines of annotated data, but the results were kind of nonesense, not better than the untrained model.Am I on the right path? Should I keep this as a self-supervised fine-tuning task?Thanks a lot for your help!
2022-03-04T08:29:29Z
juancavallotti:Then I also convert the label as aone hot vectorand keep it in thelabelsfield of my dataset.Can you tell me how you are finetuning it? The code snippet part of it.
2022-03-18T15:33:07Z
Hi!Thanks for your reply! I ended up reading the zero-shot classification paper and I realized I had the concept wrong. I ended up succeeding at my training.