Update README.md
Browse files
README.md
CHANGED
@@ -85,28 +85,7 @@ The fine-tuning dataset was compiled from the following sources:
|
|
85 |
* **Base Model:** `unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit` loaded with 4-bit quantization (`load_in_4bit=True`).
|
86 |
* **Fine-tuning Method:** Supervised Fine-Tuning (SFT) using `trl.SFTTrainer`.
|
87 |
* **Parameter Efficiency:** PEFT with LoRA (`get_peft_model`).
|
88 |
-
* `r`: 256
|
89 |
-
* `lora_alpha`: 256
|
90 |
-
* `lora_dropout`: 0.0
|
91 |
-
* `target_modules`: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
|
92 |
* **Training Configuration (`SFTConfig`):**
|
93 |
-
* `max_seq_length`: 128000
|
94 |
-
* `packing`: False
|
95 |
-
* `per_device_train_batch_size`: 4
|
96 |
-
* `gradient_accumulation_steps`: 8 (Effective Batch Size: 32)
|
97 |
-
* `warmup_ratio`: 0.02
|
98 |
-
* `num_train_epochs`: 1
|
99 |
-
* `learning_rate`: 5e-5
|
100 |
-
* `fp16`: True
|
101 |
-
* `bf16`: True (Mixed Precision Training)
|
102 |
-
* `logging_steps`: 10
|
103 |
-
* `optim`: "adamw_8bit"
|
104 |
-
* `weight_decay`: 0.01
|
105 |
-
* `lr_scheduler_type`: "cosine_with_restarts"
|
106 |
-
* `seed`: 1729
|
107 |
-
* `output_dir`: "lora_outputs_run5"
|
108 |
-
* `save_strategy`: "steps"
|
109 |
-
* `save_steps`: 1000
|
110 |
* **Optimization Kernel:** Liger kernel enabled (`use_liger=True`) for increased throughput and reduced memory usage via optimized Triton kernels for common LLM operations.
|
111 |
|
112 |
## Inference - vLLM
|
|
|
85 |
* **Base Model:** `unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit` loaded with 4-bit quantization (`load_in_4bit=True`).
|
86 |
* **Fine-tuning Method:** Supervised Fine-Tuning (SFT) using `trl.SFTTrainer`.
|
87 |
* **Parameter Efficiency:** PEFT with LoRA (`get_peft_model`).
|
|
|
|
|
|
|
|
|
88 |
* **Training Configuration (`SFTConfig`):**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
89 |
* **Optimization Kernel:** Liger kernel enabled (`use_liger=True`) for increased throughput and reduced memory usage via optimized Triton kernels for common LLM operations.
|
90 |
|
91 |
## Inference - vLLM
|