Different intermediate results given different number of epochs
We are using Hugging Face API to fine-tune a pretrained model ( BertForSequenceClassification).We see differences in the first five epochs between 5 and 15 epoch runs and do not understand why they would not be (nearly) identical given that only the number of epochs is different between those runs. ( the seed and other parameters are all the same).For example:Seed 75 epochs :,loss,learning_rate,epoch,step0,24.6558,4.955555555555556e-05,0.04,500,1,19.9439,4.9111111111111114e-05,0.09,1000,2,19.2654,4.866666666666667e-05,0.13,1500,3,20.4078,4.8222222222222225e-05,0.18,2000,4,20.3372,4.7777777777777784e-05,0.22,2500,5,20.0602,4.7333333333333336e-05,0.27,3000,6,19.6761,4.6888888888888895e-05,0.31,3500,7,20.193,4.644444444444445e-05,0.36,4000,8,19.1265,4.600000000000001e-05,0.4,4500,9,19.1949,4.555555555555556e-05,0.44,5000,10,19.5078,4.511111111111112e-05,0.49,5500,11,20.7165,4.466666666666667e-05,0.53,6000,12,20.1907,4.422222222222222e-05,0.58,6500,13,19.6967,4.377777777777778e-05,0.62,7000,14,19.6693,4.3333333333333334e-05,0.67,7500,15,20.011,4.2888888888888886e-05,0.71,8000,16,19.516,4.2444444444444445e-05,0.76,8500,17,18.9949,4.2e-05,0.8,9000,15 epochs:,loss,learning_rate,epoch,step0,18.9326,4.9851851851851855e-05,0.04,500,1,5.6773,4.970370370370371e-05,0.09,1000,2,4.6515,4.955555555555556e-05,0.13,1500,3,4.2881,4.940740740740741e-05,0.18,2000,4,3.641,4.925925925925926e-05,0.22,2500,5,3.2491,4.9111111111111114e-05,0.27,3000,6,3.012,4.896296296296297e-05,0.31,3500,7,2.8161,4.881481481481482e-05,0.36,4000,8,2.7497,4.866666666666667e-05,0.4,4500,9,2.6776,4.851851851851852e-05,0.44,5000,10,2.5254,4.837037037037037e-05,0.49,5500,11,2.6059,4.8222222222222225e-05,0.53,6000,12,2.5966,4.807407407407408e-05,0.58,6500,13,2.2252,4.792592592592593e-05,0.62,7000,14,2.3321,4.7777777777777784e-05,0.67,7500,15,2.23,4.762962962962963e-05,0.71,8000,16,2.3754,4.7481481481481483e-05,0.76,8500,Seed 0 :5 epochs:,loss,learning_rate,epoch,step0,17.7629,4.955555555555556e-05,0.04,500,1,5.6264,4.9111111111111114e-05,0.09,1000,2,4.9429,4.866666666666667e-05,0.13,1500,3,4.5756,4.8222222222222225e-05,0.18,2000,4,4.4063,4.7777777777777784e-05,0.22,2500,5,3.9688,4.7333333333333336e-05,0.27,3000,6,3.6656,4.6888888888888895e-05,0.31,3500,7,3.6779,4.644444444444445e-05,0.36,4000,8,3.2495,4.600000000000001e-05,0.4,4500,9,3.2306,4.555555555555556e-05,0.44,5000,10,3.1333,4.511111111111112e-05,0.49,5500,11,2.7543,4.466666666666667e-05,0.53,6000,12,3.1086,4.422222222222222e-05,0.58,6500,13,3.0666,4.377777777777778e-05,0.62,7000,14,3.156,4.3333333333333334e-05,0.67,7500,15,2.5553,4.2888888888888886e-05,0.71,8000,16,2.7727,4.2444444444444445e-05,0.76,8500,17,2.651,4.2e-05,0.8,9000,15 epochs:,loss,learning_rate,epoch,step0,14.8927,4.9851851851851855e-05,0.04,500,1,5.4558,4.970370370370371e-05,0.09,1000,2,4.065,4.955555555555556e-05,0.13,1500,3,3.8751,4.940740740740741e-05,0.18,2000,4,3.4581,4.925925925925926e-05,0.22,2500,5,3.1641,4.9111111111111114e-05,0.27,3000,6,2.8896,4.896296296296297e-05,0.31,3500,7,2.8967,4.881481481481482e-05,0.36,4000,8,2.5912,4.866666666666667e-05,0.4,4500,9,2.5563,4.851851851851852e-05,0.44,5000,10,2.482,4.837037037037037e-05,0.49,5500,11,2.1695,4.8222222222222225e-05,0.53,6000,12,2.447,4.807407407407408e-05,0.58,6500,13,2.4438,4.792592592592593e-05,0.62,7000,14,2.2014,4.7777777777777784e-05,0.67,7500,15,2.2,4.762962962962963e-05,0.71,8000,The only difference in the experiments is the number of epochs.We also saved the train and validation split to a file and read it from there. To make sure we are reading in the same order.My environment:transformersversion: 4.31.0Platform: Linux-4.18.0-477.15.1.el8_8.x86_64-x86_64-with-glibc2.28Python version: 3.9.6Huggingface_hub version: 0.16.4Safetensors version: 0.3.1Accelerate version: 0.21.0Here is part of my code:from transformers import (AutoTokenizer, DataCollatorWithPadding, TrainingArguments,BertForSequenceClassification, Trainer, AutoConfig)import datasetsimport numpy as npimport torchimport torch.nn as nnimport randomrandom.seed(cseed)np.random.seed(cseed)torch.manual_seed(cseed)torch.cuda.manual_seed_all(cseed)os.environ[‘CUBLAS_WORKSPACE_CONFIG’]=“:16:8”tokenizer = AutoTokenizer.from_pretrained(checkpoint, model_max_length=max_token_len)training_args = TrainingArguments(out_path,save_total_limit = 10,#load_best_model_at_end= True,report_to=None,evaluation_strategy=“steps”,eval_steps=11250,do_eval=True,num_train_epochs=epochs_num,seed = cseed)from transformers import set_seedset_seed(cseed)trian_data_from_disk = datasets.Dataset.load_from_disk(tokenized_datasets_path+“/train” , keep_in_memory=True)validation_data_from_disk = datasets.Dataset.load_from_disk(tokenized_datasets_path+“/validation” , keep_in_memory=True)model = BertForSequenceClassification.from_pretrained(checkpoint, num_labels=1)loss_fn = nn.MSELoss()trainer = CustomTrainer(model,training_args,train_dataset=trian_data_from_disk,eval_dataset=validation_data_from_disk,data_collator=data_collator,tokenizer=tokenizer,)training_results = trainer.train()