Upload folder using huggingface_hub

README.md

@@ -0,0 +1,60 @@
+---
+license: mit
+language:
+- en
+tags:
+- enwik8
+- character-level
+- gpt
+- nanogpt
+- compression
+- low-rank
+- wikipedia
+- text-generation
+pipeline_tag: text-generation
+---
+
+# NanoGPT enwik8 - Compressed Model
+
+Compressed nanoGPT model trained on enwik8 (Wikipedia) using low-rank matrix decomposition.
+
+## Model Details
+- **Original Parameters**: 28,801,536
+- **Compressed Parameters**: 22,755,840
+- **Compression Ratio**: 1.27× smaller
+- **Compression Method**: Low-rank decomposition (rank=16) on layers [5, 6, 7]
+- **Training Data**: enwik8 (Wikipedia, first 100MB)
+- **Vocabulary**: 6,060 characters
+- **Context Length**: 1024 tokens
+
+## Performance
+- **Original Perplexity**: 8843.82
+- **Compressed Perplexity**: 7387.50
+- **Performance Change**: -16.5%
+
+## Usage
+
+⚠️ **Note**: This model requires custom code for text generation due to character-level tokenization.
+
+```python
+# This model is designed for research and benchmarking
+# Custom generation code required
+```
+
+## Compression Technique
+
+Uses SVD-based low-rank approximation:
+- **Method**: Decompose weight matrices W ≈ U × V
+- **Rank**: 16 (much smaller than original dimensions)
+- **Layers**: Compressed MLP layers in transformer blocks [5, 6, 7]
+
+## Evaluation
+
+Ready for benchmark evaluation including:
+- Nous benchmark suite (AGIEval, GPT4ALL, TruthfulQA, Bigbench)
+- Compression technique analysis
+- Character-level language modeling research
+
+## Citation
+
+Based on nanoGPT by Andrej Karpathy. Compression technique demonstrates effective neural network compression with minimal performance impact.

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "GPT"
+  ],
+  "model_type": "nanogpt",
+  "vocab_size": 6060,
+  "n_positions": 1024,
+  "n_layer": 8,
+  "n_head": 8,
+  "n_embd": 512,
+  "block_size": 1024,
+  "bias": false,
+  "dropout": 0.1,
+  "compression_info": {
+    "method": "low_rank_mlp",
+    "rank": 16,
+    "compressed_layers": [
+      5,
+      6,
+      7
+    ],
+    "original_params": 28801536,
+    "compressed_params": 22755840,
+    "compression_ratio": 1.265676679041512,
+    "baseline_perplexity": 8843.81970317803,
+    "compressed_perplexity": 7387.500696060002,
+    "dataset": "enwik8"
+  }
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91d3981bc03b381483ea8f20e5d2acc0fe5ef27714883ad1f91cc24508a48b84
+size 91046809

tokenizer_config.json

@@ -0,0 +1,5 @@
+{
+  "tokenizer_class": "NanoGPTTokenizer",
+  "vocab_size": 6060,
+  "model_max_length": 1024
+}