w2v-bert-malayalam-v2

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1097
• Wer: 0.0913

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 38000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.3486	0.2859	2000	0.3181	0.4042
0.291	0.5718	4000	0.2474	0.3020
0.2196	0.8577	6000	0.2151	0.2710
0.1915	1.1437	8000	0.2131	0.2488
0.1811	1.4295	10000	0.1786	0.2204
0.1881	1.7154	12000	0.1720	0.2061
0.1598	2.0014	14000	0.1768	0.1834
0.1429	2.2873	16000	0.1741	0.1708
0.1389	2.5732	18000	0.1646	0.1560
0.1314	2.8591	20000	0.1387	0.1490
0.0953	3.1451	22000	0.1457	0.1373
0.0915	3.4310	24000	0.1287	0.1238
0.0871	3.7169	26000	0.1255	0.1145
0.0903	4.0029	28000	0.1181	0.1069
0.0723	4.2887	30000	0.1226	0.1022
0.0599	4.5746	32000	0.1115	0.0992
0.0576	4.8605	34000	0.1087	0.0977
0.0473	5.1465	36000	0.1079	0.0928
0.0485	5.4324	38000	0.1097	0.0913

Framework versions

• Transformers 4.48.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0