Upload README.md

README.md

@@ -1,3 +1,107 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+- code
+library_name: transformers
+tags:
+- causal-lm
+- moe
+- mixture-of-experts
+- qwen
+- distillation
+- svd
+- lora-merged
+- code-generation
+base_model: Qwen/Qwen3-Coder-30B-A3B-Instruct
+---
+
+# [Model Name]: A High-Fidelity Distillation of Qwen3-Coder for Advanced Code Generation
+
+## Model Description
+
+This model is a high-fidelity, distilled version of **`Qwen/Qwen3-Coder-30B-A3B-Instruct`** designed to achieve coding and reasoning capabilities approaching those of a much larger, private teacher model.
+
+It is the result of applying a sophisticated LoRA, generated via a unique distillation pipeline, and then merging those weights into the base model. The core of this process was to transfer the nuanced knowledge from a **62-layer, 160-expert teacher model** into the more efficient **48-layer, 128-expert architecture** of the `Qwen3-Coder` student model.
+
+The primary goal was to significantly enhance performance on **complex coding tasks**, where the specialized knowledge of Mixture-of-Experts (MoE) layers is critical.
+
+## The Distillation Methodology
+
+This model was not trained in a conventional sense. Instead, it was created using a layer-by-layer distillation process implemented in the `moe_distill_gpu_v15_FINAL` script. This pipeline was designed to ensure maximum precision and knowledge transfer.
+
+### Core Components
+
+*   **Teacher Model:** A private 62-layer, 160-expert/layer Qwen model.
+*   **Student Model:** `Qwen/Qwen3-Coder-30B-A3B-Instruct`.
+*   **LoRA Rank:** A high rank of **`r=2048`** was used for all modules to capture a very high degree of information from the teacher.
+
+### The Distillation Pipeline
+
+For each corresponding layer in the student and teacher, the following pipeline was executed:
+
+1.  **Spherical Linear Interpolation (SLERP):** For layers that fall between two teacher layers, SLERP was used to create a smooth, geometrically sound interpolation of the teacher's weights. This avoids the pitfalls of simple linear averaging.
+
+2.  **Singular Value Decomposition (SVD) Projection:** The core of the distillation. The (potentially blended) teacher layer's weight matrix was decomposed into its fundamental components (`U`, `S`, `V`). The **top 2048** most important components were selected and then reconstructed to fit the student layer's smaller dimensions. This high-rank projection ensures maximum fidelity.
+
+3.  **Procrustes Analysis:** After projection, the newly created "synthetic" tensor was optimally rotated in high-dimensional space to perfectly align with the student's original pre-trained tensor. This minimizes the "distance" between them before calculating the difference.
+
+4.  **DARE (Drop and Rescale):** The difference tensor (`Distilled - Aligned Student`) was then purified using DARE. This process drops a significant percentage of the lowest-magnitude values (noise) and rescales the remaining important differences, creating a clean signal for the final LoRA.
+
+### Mixture-of-Experts (MoE) Distillation
+
+The standout feature of this process is the full distillation of the MoE layers, which are critical for complex reasoning.
+
+*   **Expert Fingerprinting & Clustering:** To map the 160 teacher experts to the 128 student experts, each teacher expert was "fingerprinted." **K-Means clustering** was then used to group these 160 fingerprints into 128 distinct clusters.
+*   **Expert-to-Expert Distillation:** Each of the student's 128 experts was then distilled from a weighted blend of the teacher experts assigned to its cluster. This ensures the specialized knowledge (e.g., recursion, API usage, security patterns) is transferred.
+*   **Router Gate Distillation:** The main MoE router gate, which decides which expert to use for a given token, was also distilled to preserve the teacher's intelligent routing logic.
+
+## Intended Use
+
+This model is intended for **advanced code generation and reasoning**. It excels at tasks that require understanding complex logic, algorithms, and software architecture.
+
+*   **Primary Use:** Code generation, refactoring, explanation, and debugging.
+*   **Out of Scope:** This is not a general-purpose conversational chatbot. While it can follow instructions, its knowledge is specialized for programming tasks.
+
+## How to Use
+
+The model can be used with the standard `transformers` library pipeline. It uses the ChatML prompt format.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Make sure to replace with your Hugging Face username and the model name
+model_name = "[Your Hugging Face Username]/[Model Name]"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+
+# Example of a complex coding prompt using the ChatML format
+prompt = """
+<|im_start|>system
+You are a helpful programming assistant.
+<|im_end|>
+<|im_start|>user
+Here is a Python function that calculates Fibonacci numbers using recursion. It is very inefficient for large numbers.
+
+def fibonacci_recursive(n):
+    if n <= 1:
+        return n
+    else:
+        return fibonacci_recursive(n-1) + fibonacci_recursive(n-2)
+
+Please refactor this function to be iterative using a stack-based approach. Add comments to explain the logic.
+<|im_end|>
+<|im_start|>assistant
+"""
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.7, top_p=0.9)
+
+response = tokenizer.decode(outputs, skip_special_tokens=True)
+print(response.split("<|im_start|>assistant\n")[-1])