qnguyen3 commited on
Commit
88aac62
·
verified ·
1 Parent(s): 2ca0ff9

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -77
README.md CHANGED
@@ -4,7 +4,7 @@ base_model: Qwen/Qwen2-1.5B
4
  tags:
5
  - generated_from_trainer
6
  model-index:
7
- - name: outputs/qwen2-1.5b-super
8
  results: []
9
  ---
10
 
@@ -12,82 +12,6 @@ model-index:
12
  should probably proofread and complete it, then remove this comment. -->
13
 
14
  [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
15
- <details><summary>See axolotl config</summary>
16
-
17
- axolotl version: `0.4.1`
18
- ```yaml
19
- base_model: Qwen/Qwen2-1.5B
20
- trust_remote_code:
21
-
22
- load_in_8bit: false
23
- load_in_4bit: false
24
- strict: false
25
-
26
- datasets:
27
- - path: arcee-ai/eval_tome
28
- type: sharegpt
29
- conversation: chatml
30
- dataset_prepared_path:
31
- val_set_size: 0.0
32
- output_dir: ./outputs/qwen2-1.5b-super
33
-
34
- sequence_len: 8192
35
- sample_packing: true
36
- eval_sample_packing: false
37
- pad_to_sequence_len: true
38
-
39
- adapter:
40
- lora_model_dir:
41
- lora_r:
42
- lora_alpha:
43
- lora_dropout:
44
- lora_target_linear:
45
- lora_fan_in_fan_out:
46
-
47
- wandb_project: qwen2-1.5b
48
- wandb_entity:
49
- wandb_watch:
50
- wandb_name:
51
- wandb_log_model:
52
-
53
- gradient_accumulation_steps: 4
54
- micro_batch_size: 4
55
- num_epochs: 5
56
- optimizer: adamw_torch
57
- lr_scheduler: cosine
58
- learning_rate: 0.00005
59
-
60
- train_on_inputs: false
61
- group_by_length: false
62
- bf16: auto
63
- fp16:
64
- tf32: true
65
-
66
- gradient_checkpointing: true
67
- early_stopping_patience:
68
- resume_from_checkpoint:
69
- local_rank:
70
- logging_steps: 1
71
- xformers_attention:
72
- flash_attention: true
73
-
74
- warmup_steps: 50
75
- evals_per_epoch:
76
- saves_per_epoch: 1
77
- debug:
78
- deepspeed: zero3_bf16.json
79
- weight_decay: 0.0
80
- special_tokens:
81
- eos_token: "<|im_end|>"
82
- bos_token: "<|im_start|>"
83
-
84
- ```
85
-
86
- </details><br>
87
-
88
- # outputs/qwen2-1.5b-super
89
-
90
- This model is a fine-tuned version of [Qwen/Qwen2-1.5B](https://huggingface.co/Qwen/Qwen2-1.5B) on the None dataset.
91
 
92
  ## Model description
93
 
 
4
  tags:
5
  - generated_from_trainer
6
  model-index:
7
+ - name: qnguyen3/WitchLM-1.5B
8
  results: []
9
  ---
10
 
 
12
  should probably proofread and complete it, then remove this comment. -->
13
 
14
  [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
 
16
  ## Model description
17