Fine-tuning multilingual BERT for sequence classification with Trainer API
Hi,Im fine-tuning multilingual bert for sequence classification as this [CLS] context [SEP] choice [SEP] [PAD] …Im using the Trainer APIbatch_size = 8 # probar con 32
num_train_epochs = 6
logging_steps = len(encoded_datasets['train']) // (2 * batch_size * num_train_epochs)

training_args = TrainingArguments(
    output_dir="results",
    overwrite_output_dir=True,
    num_train_epochs=num_train_epochs,
    learning_rate=0.01, # {5e-5, 3e-5, 2e-5, 0.1}
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    warmup_steps=500,
    weight_decay=0.1, # {0, 0.01, 0.1}
    evaluation_strategy="epoch",
    save_strategy="epoch",
    logging_steps=logging_steps,
)

def compute_metrics(eval_pred):
    metric = evaluate.load("accuracy")
    logits, labels = eval_pred
    logitsTensors = torch.from_numpy(logits)
    print('eval_pred:', eval_pred)
    print('-'*40)
    print('logits:', logits)
    print('-'*40)
    print('labels:', labels)
    print('-'*40)
    probabilities = torch.softmax(logitsTensors, dim=1)
    predictions = torch.argmax(probabilities, dim=1) # [1, 0, 0, 1...] axis=-1
    print('predictions: ', predictions)
    # return {"accuracy": np.mean(predictions == labels)}
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=encoded_datasets['train'], #encoded_datasets['train'],
    eval_dataset=encoded_datasets['validation'], #encoded_datasets['validation'],
    #data_collator=data_collator,
    tokenizer=tokenizer,
    # id2label=id2label,
    # label2id=label2id,
    compute_metrics=compute_metrics
)I have experimented with different parameters but I almost always get 0.5 in accuracy and my training and validation loss stays almost the same.What could this mean?could you recommend me some guideline?Thank you!
2023-12-01T18:14:58Z
Can you share your training script?
2023-12-01T19:44:00Z
Hi, thank you for your answer.I don’t have any training script, I was hoping to train the model through Trainer as shown, without the Pytorch training loop. Is this approach correct?accuracy always stuck in 50%, what could that mean?
2023-12-05T22:56:57Z
can you try this and see what is your accuracy coming out to be?logits, labels = p
    logits = logits.tolist()
    labels = labels.tolist()
    pred = np.argmax(logits, axis=1).tolist()
    
    logits_tensor = torch.tensor(logits)
    prob = torch.nn.functional.softmax(logits_tensor, dim=-1).tolist()
    accuracy = accuracy_score(y_true=labels, y_pred=pred)
2023-12-07T16:57:45Z
hi, thank you for your help.I ran the experiment for 6 epochs, these are the results:{‘eval_loss’: 0.6945369243621826, ‘eval_accuracy’: 0.4765625, ‘eval_runtime’: 1.1837, ‘eval_samples_per_second’: 108.136, ‘eval_steps_per_second’: 13.517, ‘epoch’: 1.0}{‘eval_loss’: 0.6935781836509705, ‘eval_accuracy’: 0.5078125, ‘eval_runtime’: 1.1774, ‘eval_samples_per_second’: 108.718, ‘eval_steps_per_second’: 13.59, ‘epoch’: 2.0}{‘eval_loss’: 0.6960821747779846, ‘eval_accuracy’: 0.5, ‘eval_runtime’: 1.1887, ‘eval_samples_per_second’: 107.677, ‘eval_steps_per_second’: 13.46, ‘epoch’: 3.0}{‘eval_loss’: 0.6931933760643005, ‘eval_accuracy’: 0.5, ‘eval_runtime’: 1.1842, ‘eval_samples_per_second’: 108.091, ‘eval_steps_per_second’: 13.511, ‘epoch’: 4.0}{‘eval_loss’: 0.6936441659927368, ‘eval_accuracy’: 0.484375, ‘eval_runtime’: 1.183, ‘eval_samples_per_second’: 108.197, ‘eval_steps_per_second’: 13.525, ‘epoch’: 5.0}{‘eval_loss’: 0.6934036016464233, ‘eval_accuracy’: 0.484375, ‘eval_runtime’: 1.1699, ‘eval_samples_per_second’: 109.411, ‘eval_steps_per_second’: 13.676, ‘epoch’: 6.0}{‘train_runtime’: 217.6431, ‘train_samples_per_second’: 31.758, ‘train_steps_per_second’: 3.97, ‘train_loss’: 0.7024143382355019, ‘epoch’: 6.0}There is no any change in evaluation loss, and accuracy is stuck. I also notice that all logits are negatives as this (why is that):logits_tensor: tensor([[-0.0935, -0.1828],[-0.1098, -0.1829],[-0.0728, -0.1879],[-0.0940, -0.1868],[-0.2301, -0.2504],[-0.2266, -0.2678],[-0.2459, -0.2656],[-0.2376, -0.2704],[-0.1885, -0.0431],[-0.2165, -0.0214],…I think my validation partition is wrong formulated. I believe I have to transform back to the original format and maps the highest logit to the corresponding label which would be the prediction.
2023-12-08T16:32:35Z
Maybe your data quality is bad. i keep getting 40% accuracy also so i improved my data now i am at 60%. how big is your training data? You will have to do an error analysis, make confusion matrix and start with the maximum classes being confused with each other.
2023-12-08T17:47:50Z
Hi there,Im fine-tuning a spanish version of bert with 1152 instances. I changed the training now with native Pytorch. I will explore in deep what you are telling me. I also believe i’m doing the evaluation wrong.I am evaluating sentences for two options/candidates as a binary classification task, where my classes are 0 and 1.My sentences transform in two with every option. I think I need to transform the sentences back to the original with the predicted label of the model, and evaluate in that validation set.I do appreciate your advise
2023-12-12T00:53:21Z
mister, can we talk about our experiments and share knowledges about LLMs and programming?this is my email: musta.ali.saba @ g mail . com