How do I fine-tune roberta-large for text classification
Hi there,I have been doing the HF course and decided to apply what I have learned but I have unfortunately encountered some errors at the model.fit() stage.I extracted BBC text data as an excel file from kaggle and converted it to a DatasetDict as below:image1132×612 32.7 KBLoaded the tokenizer and tokenized the text featuresimage927×583 33 KBTrain_test_val splitimage1242×749 53.3 KBConverted my data to tf_data and padded with DataCollator and instantiated the modelimage1610×1108 88.6 KBOptimizer and compile:image1584×616 54.1 KBGEtting the error at the below stageimage1584×1083 160 KBNot sure what I am doing wrong here as I tried following the steps in the course, thanks in advance
2021-12-15T15:10:14Z
Managed to get it to work by changing ‘label’ to “labels” but now I have a different error during model.fitlabels.shape is inconsistent with logits.shapeimage1672×1025 144 KB
2021-12-16T02:29:17Z
@lewtunare you able to assist please?
2021-12-16T09:53:52Z
Hi,It depends on what loss function the model has defined. As explained onStackoverflow, your labels must either be 1-dimensional or 2-dimensional, depending on the loss function being used.i.e. they must either be of shape (batch_size,) in which case they contain the class indices, or of shape (batch_size, num_labels), in which case they contain one-hot encoded targets.
2021-12-16T10:05:07Z
@Rocketknight1
2021-12-16T16:43:53Z
Hi@nickmuchi, the key is in the warning that pops up when youcompile()! When you compile without specifying aloss, the model will compute loss internally. For this to work, though, the labels need to be in your input dict. We talk about this in theDebugging your Training Pipelinesection of the course.There are two solutions here. One is to change your calls toto_tf_dataset(). Instead ofcolumns=["attention_mask", "input_ids"],
label_cols=["labels"]docolumns=["attention_mask", "input_ids", "labels"],This will put your labels in the input dict, and the model will be able to compute a loss in the forward pass. This is simple, but might cause issues with your accuracy metric. The alternative option is to leave the labels where they are, but instead to use a proper Keras loss. In that case, you would leave the call toto_tf_dataset()unchanged, but change yourcompile()call tomodel.compile(
optimizer=optimizer, 
metrics=['accuracy'],
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
)That should work, and will allow you to keep using theaccuracymetric too. Let me know if you encounter any other problems!
2021-12-16T22:35:24Z
Rocketknight1:tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)Thanks for your response, so I tried you suggestion but still getting the error and to be clear my labels are NOT one hot encoded.image1586×725 70.6 KBimage1486×1099 126 KBimage1604×714 106 KBWeird thing is that it worked when, instead of using tf_to_dataset, I used tf_from_tensor_slices. Wanted to use the former as I was following along that part of the courseimage1595×332 25.5 KBappreciate your time and help
2021-12-17T10:48:05Z
sorry I completely misunderstood what you said, reread and implemented and it worked!!! thank you!