mistral_sky_o1_5_full

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.1 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.5358

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.7175	0.0835	100	0.6957
0.679	0.1669	200	0.6798
0.6507	0.2504	300	0.6593
0.6291	0.3339	400	0.6436
0.6013	0.4174	500	0.6191
0.5802	0.5008	600	0.6022
0.5516	0.5843	700	0.5789
0.5256	0.6678	800	0.5654
0.5129	0.7513	900	0.5521
0.4936	0.8347	1000	0.5406
0.4851	0.9182	1100	0.5366

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.0
• Tokenizers 0.19.1