The Distillation Research: Why 15,000 Clean Traces Outperform 100,000 Raw Ones

By Siddharth N.R. — Open-Source AI/ML Researcher, Pluto AI Research Lab

The AI industry is currently obsessed with scale.

Every week, a new foundational model arrives with increasingly enormous parameter counts, longer context windows, and higher benchmark scores. Models such as Moonshot's Kimi-K3 represent the extreme end of this trend, with 2.8 trillion parameters.

But as a developer building real-world AI applications, I am interested in the opposite question:

How do we take frontier-model intelligence and compress it into a 3B parameter model that can run on a consumer laptop?

The answer is knowledge distillation.

But over the last two weeks, I discovered something important:

Raw distillation is a trap.

If you do not rigorously clean your teacher-model traces, you are not necessarily teaching your student model to become smarter. You may be teaching it to become lazy.

This is the story of the Atlas-Frontier-Distill Project — an empirical study into the illusion of improvement, the anatomy of low-quality AI traces, and the MLOps pipeline required to turn noisy frontier-model outputs into high-signal training data.

---

Part 1: The Mechanics of Knowledge Distillation

In traditional machine learning, Knowledge Distillation (KD) is a process where a smaller student model learns to approximate the probability distributions produced by a much larger teacher model.

In modern Large Language Models, the process often looks different.

Because we generally do not have access to the raw logits of closed-source frontier models, behavioral distillation can instead operate through text traces.

The process is relatively straightforward:

1. Prompt a large teacher model with thousands of problems.
2. Collect its responses.
3. Filter the resulting traces.
4. Fine-tune a smaller student model on the resulting dataset.
5. Evaluate whether the student has acquired useful behaviors from the teacher.

The hypothesis is simple:

A sufficiently capable teacher can transfer useful reasoning patterns, coding strategies, formatting behavior, and problem-solving approaches into a much smaller student model.

For this project, I aggregated raw coding and debugging traces from three frontier models:

• Kimi-K3
• GPT-5.6-Sol
• Fable-5

The goal was to distill their agentic coding capabilities into Qwen2.5-Coder-3B-Instruct, producing a highly efficient coding model suitable for edge deployment.

However, auditing the raw data revealed a major problem.

---

Part 2: The "Lazy Trace" Problem

When researchers collect large-scale synthetic datasets, success is often measured by volume:

"We collected 100,000 coding traces."

The implicit assumption is:

$$\text{More Data}\Rightarrow \text{Better Student Model}\backslash text\{More\; Data\}\; \backslash Rightarrow\; \backslash text\{Better\; Student\; Model\}$$

But this relationship breaks down when a significant percentage of the data contains failures, incomplete executions, hallucinations, or low-effort responses.

API failures, rate limits, tool-execution failures, incomplete agent trajectories, and model refusals can all introduce noise into a distillation dataset.

I audited 46,875 raw rows and discovered that more than 56% contained what I classified as lazy failures or otherwise low-signal traces.

What Does a Lazy Trace Look Like?

Consider this example from the Kimi-K3 dataset.

The original prompt was a complex agentic code-review instruction. The model was expected to:

• inspect repository state,
• use shell tools,
• compare files,
• inspect the implementation,
• execute tests,
• and return a strict JSON verdict.

Instead, the model produced:

{
  "verdict": "needs_human",
  "requirements": [],
  "summary": "I’ll independently inspect repository state, compare the protected test byte-for-byte, read the complete shipped patch and implementation, then run the specified test command once."
}

At first glance, the response looks structured and professional.

But there is a fundamental problem:

The model did not actually perform the requested work.

It merely described what it intended to do.

There was:

• no repository inspection,
• no shell execution,
• no code review,
• no test execution,
• no actual analysis,
• and no useful requirements.

The response effectively says:

"I will do the work."

without actually doing the work.

This distinction matters enormously during distillation.

If a student model is trained on thousands of traces like this, it may learn an undesirable behavioral policy:

$$\text{Hard Problem}\to \text{Promise to Investigate}\to \text{Give Up}\backslash text\{Hard\; Problem\}\; \backslash rightarrow\; \backslash text\{Promise\; to\; Investigate\}\; \backslash rightarrow\; \backslash text\{Give\; Up\}$$

instead of:

$$\text{Hard Problem}\to \text{Investigate}\to \text{Reason}\to \text{Solve}\backslash text\{Hard\; Problem\}\; \backslash rightarrow\; \backslash text\{Investigate\}\; \backslash rightarrow\; \backslash text\{Reason\}\; \backslash rightarrow\; \backslash text\{Solve\}$$

This is what I call the Distillation Bottleneck.

Raw dataset scale does not necessarily increase the amount of useful knowledge transferred to the student.

Sometimes, it simply increases the amount of noise.

---

Part 3: The Data-Centric AI Pipeline

To address this problem, I built a multi-stage data-processing pipeline focused on signal density rather than dataset volume.

The pipeline consisted of three major stages:

1. Raw data extraction
2. Schema normalization
3. Quality filtering

---

Stage 1: The Pandas Bypass

The three source datasets were formatted differently.

They contained combinations of:

• ShareGPT-style conversations
• OpenAI Messages format
• Raw prompt/completion pairs
• Nested conversation structures
• Different field names
• Missing or null values

In addition, the Hugging Face datasets library encountered schema conflicts in one of the repositories, including string-to-null casting issues.

Instead of spending time forcing incompatible schemas through the loader, I bypassed the abstraction layer.

I loaded the raw .parquet and .jsonl files directly using Pandas.

This gave me complete control over:

• type coercion,
• missing values,
• nested structures,
• schema normalization,
• filtering,
• and data validation.

The objective was simple:

Recover the maximum amount of usable information before filtering.

---

Stage 2: Aggressive Schema Unification

I implemented a custom extraction layer that checked for multiple possible representations of the same conversational structure.

For example, the pipeline searched for fields such as:

prompt
instruction
conversations
messages

Each source was then transformed into a consistent conversational representation.

The final training format was normalized around ChatML-style conversations:

<|im_start|>system
...
<|im_end|>

<|im_start|>user
...
<|im_end|>

<|im_start|>assistant
...
<|im_end|>

This normalization step was critical because a distillation dataset is only useful if the student receives consistent training signals.

---

Stage 3: The Quality Filter

This was the most important stage of the pipeline.

Instead of asking:

"How can I keep as much data as possible?"

I asked:

"How much low-quality behavior can I remove while preserving useful reasoning?"

I implemented several heuristic filters.

1. Length Check

Responses below 50 characters were removed.

The assumption was that extremely short assistant responses were unlikely to contain meaningful coding reasoning or implementation details.

2. Failure Keyword Detection

Specific failure patterns were identified.

For example:

"verdict": "needs_human"

combined with:

"requirements": []

was treated as a strong signal of an incomplete agentic trace.

3. Promise Detection

The pipeline also identified traces where the model claimed it would perform an action rather than actually performing it.

Examples included patterns such as:

"I'll inspect..."
"I will check..."
"I'll review..."
"I'll run..."
"I'll investigate..."

when those statements were not followed by evidence of the corresponding action.

The goal was not to remove every sentence containing these phrases.

The goal was to identify traces where promised work replaced completed work.

---

The Dataset Transformation

The result was significant.

After filtering, I was left with exactly:

15,746 high-signal coding traces.

The resulting dataset was approximately 36MB of dense Parquet text, compared with more than 1GB of raw source data.

I named the resulting dataset:

Atlas-Frontier-Model-Traces

and open-sourced it for the community.

---

Part 4: Resilient Training on Ephemeral Compute

Once the dataset was cleaned, the next challenge was training.

The target model was a 3B parameter coding model, and the training environment was Kaggle with a Tesla T4 GPU.

Training was expected to take approximately 9.5 hours, while Kaggle imposes session limits.

That creates a significant reliability problem.

If training runs for eight hours and the session terminates, a conventional training pipeline can lose substantial progress.

I therefore designed a 3-layer cross-session checkpoint recovery system.

---

Layer 1: Local Checkpoints

The trainer periodically saved the training state to the Kaggle local filesystem.

This included:

optimizer.pt
scheduler.pt
adapter_model.safetensors
trainer_state.json

Checkpoints were created every 100 steps.

---

Layer 2: Hugging Face Hub Backup

The checkpoint directory was automatically pushed to a private Hugging Face repository.

This created a persistent remote copy of the training state.

Therefore:

$$\text{Local Failure}\Rightarrow ̸\text{Training Loss}\backslash text\{Local\; Failure\}\; \backslash not\backslash Rightarrow\; \backslash text\{Training\; Loss\}$$

If Kaggle wiped the local environment, the checkpoint remained available remotely.

---

Layer 3: Dynamic Resume

When the training environment restarted, the pipeline queried the Hugging Face Hub for the latest checkpoint.

It then:

1. Located the latest checkpoint.
2. Downloaded the trainer state.
3. Read trainer_state.json.
4. Extracted the latest global_step.
5. Restored the model and optimizer state.
6. Resumed training from the previous checkpoint.

The objective was to make the training environment effectively stateless.

The compute instance could disappear.

The training process would survive.

---

Training Configuration

The final training run used:

• Base Model: Qwen2.5-Coder-3B-Instruct
• Method: QLoRA
• Quantization: 4-bit NF4
• LoRA Rank: 32
• Optimizer: paged_adamw_8bit
• Training Steps: 493
• Final Training Loss: 1.7056
• Hardware: Tesla T4
• Dataset: Atlas-Frontier-Model-Traces

The combination of QLoRA and an 8-bit paged optimizer helped keep VRAM usage manageable while training the 3B parameter model.

The final model was released as:

Atlas-Frontier-Distill-3B

---

Part 5: The Result and Behavioral Evaluation

A major problem with evaluating distillation is that conventional benchmarks do not necessarily capture behavioral changes.

Benchmarks such as MMLU and HumanEval can tell us whether a model can solve a particular class of problems.

But they do not necessarily answer questions such as:

• Did the model become more verbose?
• Did it retain instruction-following behavior?
• Did the model start adding unnecessary preambles?
• Did JSON formatting become less reliable?
• Did reasoning consistency change?
• Did the model acquire undesirable teacher behaviors?

For this reason, I evaluated the model using my own open-source CLI tool:

llm-diff

Think of it as:

Git diff for LLM behavior.

The tool compares model outputs and measures behavioral differences across controlled prompts.

---

Behavioral Delta

I compared:

Qwen2.5-Coder-3B-Instruct

against:

Atlas-Frontier-Distill-3B

The resulting measurements showed:

Instruction Fidelity

1.00

The distilled model maintained strict instruction-following behavior.

Verbosity Profile

Frontier-style filler preambles such as:

"Certainly! Here is..."
"Sure! Let's..."
"Of course..."

were reduced to effectively 0 in the evaluated prompts.

The distilled traces encouraged the model to move more directly toward the requested output.

Reasoning Consistency

1.00

The model maintained logical consistency across the evaluated reframed syllogism prompts.

These results suggest that the distillation process transferred useful behavioral characteristics without substantially degrading the base model's instruction-following capabilities.

---

The Core Finding

The most important observation from this experiment is not the final model size.

It is the relationship between dataset quality and effective training signal.

Consider two hypothetical datasets:

$$D_{raw}=100,000\text{ traces}D\_\{raw\}\; =\; 100,000\; \backslash text\{\; traces\}$$

and:

$$D_{clean}=15,000\text{ high-signal traces}D\_\{clean\}\; =\; 15,000\; \backslash text\{\; high-signal\; traces\}$$

If a large fraction of (D_{raw}) consists of incomplete, failed, repetitive, or lazy trajectories, then the effective information content can be substantially lower than its row count suggests.

In other words:

$$\text{Dataset Size}\mathrm{\ne }\text{Dataset Information}\backslash text\{Dataset\; Size\}\; \backslash neq\; \backslash text\{Dataset\; Information\}$$

A better approximation is:

$$\text{Effective Training Signal}================================\text{Data Volume}\times \text{Signal Density}\backslash text\{Effective\; Training\; Signal\}\; ================================\; \backslash text\{Data\; Volume\}\; \backslash times\; \backslash text\{Signal\; Density\}$$

This changes how we should think about synthetic distillation datasets.

---

Conclusion: Data Quality > Compute Scale

The Atlas-Frontier-Distill project reinforced a lesson I believe will become increasingly important as smaller models become more capable:

Intelligence transfer is not simply a function of how much data you collect. It depends heavily on the quality of the behavior you choose to teach.

I started with:

46,875 raw traces

and ended with:

15,746 high-signal traces.

More than 20,000 rows were removed.

The resulting dataset occupied only around 36MB, yet it became the foundation for distilling frontier-model coding behavior into a 3B parameter model designed for efficient local deployment.

The broader lesson is straightforward:

Do not blindly maximize the number of synthetic traces. Maximize the amount of useful behavior contained within them.

If you are building a distillation dataset, audit your teacher traces.

Look for:

• incomplete agent trajectories,
• tool-use failures,
• hallucinated actions,
• empty outputs,
• repetitive responses,
• refusal patterns,
• placeholder reasoning,
• and promises to perform work that was never actually performed.

Then remove them.

Because in knowledge distillation:

$$\boxed{{\displaystyle \text{High-Signal Data}>\text{Raw Data Volume}}}\backslash boxed\{\backslash text\{High-Signal\; Data\}\; >\; \backslash text\{Raw\; Data\; Volume\}\}$$

The future of efficient AI may not simply be about building larger models.

It may be about becoming much better at deciding what smaller models should learn.

---

Open-Source Assets

Model

Siddh07ETH/Atlas-Frontier-Distill-3B

Dataset

Siddh07ETH/Atlas-Frontier-Model-Traces

Evaluation Tool

Pluto-AI-Labs/llm-diff

Install:

pip install pluto-llm-diff

---

Final Takeaway

15,000 clean traces can be more valuable than 100,000 raw traces.

Not because fewer examples are inherently better.

Because every training example is a behavioral instruction.

If the teacher produces useful reasoning, the student can learn it.

If the teacher produces lazy behavior, the student can learn that too.

Distillation is not just compression.

It is behavioral selection.