lucifertrj commited on
Commit
1f713ac
·
verified ·
1 Parent(s): 50f2e6c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -21
README.md CHANGED
@@ -85,28 +85,7 @@ The fine-tuning dataset was compiled from the following sources:
85
  * **Base Model:** `unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit` loaded with 4-bit quantization (`load_in_4bit=True`).
86
  * **Fine-tuning Method:** Supervised Fine-Tuning (SFT) using `trl.SFTTrainer`.
87
  * **Parameter Efficiency:** PEFT with LoRA (`get_peft_model`).
88
- * `r`: 256
89
- * `lora_alpha`: 256
90
- * `lora_dropout`: 0.0
91
- * `target_modules`: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
92
  * **Training Configuration (`SFTConfig`):**
93
- * `max_seq_length`: 128000
94
- * `packing`: False
95
- * `per_device_train_batch_size`: 4
96
- * `gradient_accumulation_steps`: 8 (Effective Batch Size: 32)
97
- * `warmup_ratio`: 0.02
98
- * `num_train_epochs`: 1
99
- * `learning_rate`: 5e-5
100
- * `fp16`: True
101
- * `bf16`: True (Mixed Precision Training)
102
- * `logging_steps`: 10
103
- * `optim`: "adamw_8bit"
104
- * `weight_decay`: 0.01
105
- * `lr_scheduler_type`: "cosine_with_restarts"
106
- * `seed`: 1729
107
- * `output_dir`: "lora_outputs_run5"
108
- * `save_strategy`: "steps"
109
- * `save_steps`: 1000
110
  * **Optimization Kernel:** Liger kernel enabled (`use_liger=True`) for increased throughput and reduced memory usage via optimized Triton kernels for common LLM operations.
111
 
112
  ## Inference - vLLM
 
85
  * **Base Model:** `unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit` loaded with 4-bit quantization (`load_in_4bit=True`).
86
  * **Fine-tuning Method:** Supervised Fine-Tuning (SFT) using `trl.SFTTrainer`.
87
  * **Parameter Efficiency:** PEFT with LoRA (`get_peft_model`).
 
 
 
 
88
  * **Training Configuration (`SFTConfig`):**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
  * **Optimization Kernel:** Liger kernel enabled (`use_liger=True`) for increased throughput and reduced memory usage via optimized Triton kernels for common LLM operations.
90
 
91
  ## Inference - vLLM