Finetune xlm roberta base(overfitting ,any solution )
import pandas as pdfrom sklearn.model_selection import train_test_splitimport torchfrom torch.utils.data import Dataset, DataLoaderfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments,DataCollatorWithPaddingLoad your datasetdf = pd.read_excel(‘dataset_final.xlsx’)Split the dataset into training and testing setstrain_df, test_df = train_test_split(df, test_size=0.2, random_state=42)Create a PyTorch Datasetclass CustomDataset(Dataset):definit(self, encodings, labels):self.encodings = encodingsself.labels = labelsdef __getitem__(self, idx):
    item = {key: val[idx] for key, val in self.encodings.items()}
    item['labels'] = torch.tensor(self.labels[idx])
    return item

def __len__(self):
    return len(self.labels)tokenizer = AutoTokenizer.from_pretrained(“xlm-roberta-base”)data_collator = DataCollatorWithPadding(tokenizer=tokenizer)train_encodings = tokenizer(list(train_df[‘text’]), truncation=True, padding=True, return_tensors=‘pt’)test_encodings = tokenizer(list(test_df[‘text’]), truncation=True, padding=True, return_tensors=‘pt’)Prepare the labelstrain_labels = list(train_df[‘label’])test_labels = list(test_df[‘label’])Create DataLoader objectstrain_dataset = CustomDataset(train_encodings, train_labels)test_dataset = CustomDataset(test_encodings, test_labels)import evaluateaccuracy = evaluate.load(“accuracy”)import numpy as npdef compute_metrics(eval_pred):predictions, labels = eval_predpredictions = np.argmax(predictions, axis=1)return accuracy.compute(predictions=predictions, references=labels)id2label = {0: “negative”, 1: “neutral”,2:“positive”}label2id = {“negative”: 0, “neutral”: 1,“positive”:2}model = AutoModelForSequenceClassification.from_pretrained(“xlm-roberta-base”,hidden_dropout_prob=0.3, attention_probs_dropout_prob=0.25, num_labels=3, id2label=id2label, label2id=label2id)from transformers import EarlyStoppingCallbacktraining_args = TrainingArguments(output_dir=“testing_finetune”,learning_rate=2e-5,per_device_train_batch_size=16,per_device_eval_batch_size=16,num_train_epochs=3.0,warmup_steps=600,weight_decay=0.1,evaluation_strategy=“epoch”,save_strategy=“epoch”,load_best_model_at_end=True,push_to_hub=True,)Create and configure your Trainertrainer = Trainer(model=model,args=training_args,train_dataset=train_dataset,eval_dataset=test_dataset,tokenizer=tokenizer,data_collator=data_collator,compute_metrics=compute_metrics,)trainer.train()trainer.push_to_hub()this is my code ,can someone help me? my result is always over fitting . thank you
2023-12-22T03:54:25Z
Can we get some additional information? You’re training for 3 epochs with 600 warmup steps with a batch size of 16, so you’re doing warmup on 600x16=9600 sequences - is your dataset sufficiently large for this?Also, when is it overfitting? Is it already overfit after the first epoch? Or only with the final model? If you can provide train/test loss curves or eval info per epoch that would be a big help in identifying the problem.If it’s already overfitting after the first epoch, that could be an indication that perhaps your train and test splits aren’t representative of one another. Did you shuffle the dataset properly before creating these splits? Maybe the train set has a bunch of duplicates or near-duplicates, making it harder for the model to generalize? Also, what other hyperparameters have you tried? Did those improve/worsen your results?
2023-12-26T04:29:48Z
for now i am using 3 epoch with no warmup ,for right now ,and my train dataset is 29,306 and my test is 7,328training_args = TrainingArguments(output_dir=“TESTING-ROBERTA”,learning_rate=2e-5,per_device_train_batch_size=16,per_device_eval_batch_size=16,num_train_epochs=3,weight_decay=0.2,evaluation_strategy=“epoch”,save_strategy=“epoch”,load_best_model_at_end=True,logging_steps=100,save_total_limit=3,push_to_hub=True,)trainer = Trainer(model=model,args=training_args,train_dataset=tokenized_malay[“train”],eval_dataset=tokenized_malay[“test”],tokenizer=tokenizer,data_collator=data_collator,compute_metrics=compute_metrics,)trainer.train()this is my new trainner argummentTraining LossEpochStepValidation LossAccuracy0.63351.018320.66390.72500.51222.036640.66120.74040.38873.054960.71920.7389this is my new result
2023-12-26T04:30:59Z
i did shuffle the dataset and split into test_size =0.2