Training in progress, step 5000

README.md

@@ -1,7 +1,7 @@
 ---
 base_model: gpt2
 datasets:
-- distily/synth_gpt2_t1_seq_1M
+- togethercomputer/RedPajama-Data-V2
 library_name: Distily
 license: creativeml-openrail-m
 tags:
@@ -18,7 +18,7 @@ model-index:
 
 Distilled with [Distily](https://github.com/lapp0/distily) library
 using teacher model [gpt2](https://huggingface.co/gpt2)
-on dataset [distily/synth_gpt2_t1_seq_1M](https://huggingface.co/datasets/distily/synth_gpt2_t1_seq_1M).
+on dataset [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2).
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -78,13 +78,13 @@ GPT2LMHeadModel(
 
 # Resource Usage
 
-- Max Train VRAM Use: 15.7135 GB
-- Available VRAM: 23.4329 GB
+- Max Train VRAM Use: 15.7182 GB
+- Available VRAM: 23.6429 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
-- CPUs: 64
-- CPU Memory: 251.7299 GB
-- CPU Memory Bandwidth: 1600 GB/s
+- CPUs: 32
+- CPU Memory: 61.9353 GB
+- CPU Memory Bandwidth: 800 GB/s
 
 # Distillation (Teacher -> Student) Architecture Difference:
 
@@ -115,10 +115,10 @@ GPT2LMHeadModel(
 <br/>
 
 # Train Dataset
-Trained on 681,027,436 tokens from the [distily/synth_gpt2_t1_seq_1M](https://huggingface.co/datasets/distily/synth_gpt2_t1_seq_1M) dataset.
+Trained on 640,432,662 tokens from the [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) dataset.
 
 - Num Samples: `998,000`
-- Subset: `None`
+- Subset: `sample`
 - Split: `train`
 
 
@@ -154,7 +154,7 @@ The following hyperparameters were used during training:
 - eval_batch_size: `8`
 - seed: `42`
 - optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08`
-- lr_scheduler_type: `constant`
+- lr_scheduler_type: `polynomial`
 - num_epochs: `1.0`
 - distillation_objective: `DistillationObjective(
     logits_loss_component=LossComponent(
@@ -172,7 +172,7 @@ The following hyperparameters were used during training:
         projector='orthogonal'
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7a7afdb74e80>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f3ee68c4be0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `distilbert/distilgpt2`
 - student_model_config: `None`
@@ -182,14 +182,15 @@ The following hyperparameters were used during training:
 - teacher_model_name_or_path: `gpt2`
 - teacher_load_in_8bit: `False`
 - teacher_load_in_4bit: `False`
-- dataset_uri: `distily/synth_gpt2_t1_seq_1M`
-- dataset_subset: `None`
+- dataset_uri: `togethercomputer/RedPajama-Data-V2`
+- dataset_subset: `sample`
 - dataset_split: `train`
-- dataset_column_name: `text`
+- dataset_column_name: `raw_content`
 - dataset_sample_size: `1000000`
 - dataset_test_size: `0.002`
-- dataset_shuffle: `True`
+- dataset_shuffle: `False`
 - dataset_shuffle_seed: `42`
+- dataset_trust_remote_code: `False`
 - gradient_accumulation_steps: `1`
 - weight_decay: `0.0`
 - max_grad_norm: `1.0`

logs/dataset_split=train, dataset_subset=None, dataset_trust_remote_code=True, dataset_uri=Skylion007_openwebtext/events.out.tfevents.1725796463.0b434856d812

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:606b5b78386d438a7281ba5cd927acbbab6ddc144b9e5db6e7f7c7af76de85b7
+size 162094

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a1a292cdeff20913a9e61397f7e618e7519c47cb9bf60bbc8759459422b0bfaa
+oid sha256:19233e3bb76d887efe22c96f07164d9798fe33d4a24377496890a382f4e5373a
 size 163832792

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:df27e837bc207577c331df3b67b7c8b3301c875f04f12f54132150840661b885
-size 5560
+oid sha256:b47ff358d893d714321d24fe3a57959efbf53fc8124c51a71a9cd91ef56abe87
+size 5496