codebert-fine-tuned

This model is a fine-tuned version of microsoft/codebert-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.0908

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
3.5941	0.0325	500	2.0780
2.091	0.0651	1000	1.8173
1.9005	0.0976	1500	1.6783
1.7817	0.1301	2000	1.6071
1.712	0.1626	2500	1.5634
1.6661	0.1952	3000	1.5229
1.6348	0.2277	3500	1.4965
1.6106	0.2602	4000	1.4514
1.5685	0.2928	4500	1.4360
1.5419	0.3253	5000	1.4203
1.5429	0.3578	5500	1.4026
1.5069	0.3903	6000	1.3959
1.5021	0.4229	6500	1.3819
1.4651	0.4554	7000	1.3660
1.4704	0.4879	7500	1.3544
1.4799	0.5205	8000	1.3428
1.44	0.5530	8500	1.3357
1.4433	0.5855	9000	1.3224
1.4297	0.6180	9500	1.3173
1.4115	0.6506	10000	1.3069
1.4119	0.6831	10500	1.2996
1.3908	0.7156	11000	1.2972
1.4022	0.7482	11500	1.2879
1.381	0.7807	12000	1.2843
1.374	0.8132	12500	1.2747
1.382	0.8457	13000	1.2734
1.3746	0.8783	13500	1.2576
1.3724	0.9108	14000	1.2605
1.3404	0.9433	14500	1.2560
1.3452	0.9759	15000	1.2414
1.3433	1.0084	15500	1.2373
1.3273	1.0409	16000	1.2398
1.3175	1.0735	16500	1.2311
1.3123	1.1060	17000	1.2217
1.3095	1.1385	17500	1.2213
1.3229	1.1710	18000	1.2167
1.2995	1.2036	18500	1.2185
1.3019	1.2361	19000	1.2144
1.299	1.2686	19500	1.2093
1.2784	1.3012	20000	1.1990
1.2886	1.3337	20500	1.2032
1.2788	1.3662	21000	1.1943
1.284	1.3987	21500	1.1975
1.2706	1.4313	22000	1.1878
1.2771	1.4638	22500	1.1856
1.2731	1.4963	23000	1.1797
1.2607	1.5289	23500	1.1919
1.2729	1.5614	24000	1.1872
1.272	1.5939	24500	1.1712
1.251	1.6264	25000	1.1656
1.2437	1.6590	25500	1.1665
1.2523	1.6915	26000	1.1697
1.2393	1.7240	26500	1.1546
1.2521	1.7566	27000	1.1595
1.2498	1.7891	27500	1.1541
1.2187	1.8216	28000	1.1586
1.2311	1.8541	28500	1.1530
1.2419	1.8867	29000	1.1412
1.2246	1.9192	29500	1.1460
1.2381	1.9517	30000	1.1475
1.2237	1.9843	30500	1.1432
1.2273	2.0168	31000	1.1458
1.2167	2.0493	31500	1.1368
1.2039	2.0818	32000	1.1358
1.2142	2.1144	32500	1.1410
1.2003	2.1469	33000	1.1278
1.2052	2.1794	33500	1.1344
1.2094	2.2120	34000	1.1378
1.2128	2.2445	34500	1.1291
1.1936	2.2770	35000	1.1280
1.195	2.3095	35500	1.1278
1.207	2.3421	36000	1.1220
1.1969	2.3746	36500	1.1248
1.188	2.4071	37000	1.1159
1.1921	2.4397	37500	1.1187
1.1916	2.4722	38000	1.1196
1.1797	2.5047	38500	1.1167
1.1865	2.5372	39000	1.1135
1.1787	2.5698	39500	1.1154
1.1865	2.6023	40000	1.1174
1.1754	2.6348	40500	1.1161
1.1805	2.6674	41000	1.1085
1.1786	2.6999	41500	1.1116
1.1689	2.7324	42000	1.1069
1.1755	2.7649	42500	1.1032
1.1858	2.7975	43000	1.1027
1.1722	2.8300	43500	1.1027
1.1686	2.8625	44000	1.1002
1.1801	2.8951	44500	1.1016
1.1596	2.9276	45000	1.1024
1.1788	2.9601	45500	1.1052
1.1609	2.9926	46000	1.0908

Framework versions

• Transformers 4.46.3
• Pytorch 2.4.1+cu121
• Datasets 3.1.0
• Tokenizers 0.20.3