llm-diff: Behavioral Regression Testing for Large Language Models
Intelligence isn't about scale. It's about precision.
Abstract
Large language models are evolving rapidly. New checkpoints, fine-tunes, quantizations, and API releases appear continuously, and conventional benchmarks make it easy to measure whether a model's score has improved.
But benchmark improvement does not necessarily mean behavioral improvement.
A model can achieve a higher benchmark score while becoming more verbose, violating formatting constraints, or producing less consistent reasoning. These changes can remain invisible until the model reaches production.
To address this gap, I built llm-diff — an open-source command-line tool for behavioral regression testing of language models.
The central idea is simple:
If
git diffshows what changed in code,llm-diffshows what changed in model behavior.
The project is designed to provide a lightweight evaluation layer that can run before a model is promoted into production or integrated into a larger MLOps pipeline.
1. The Problem
Most model evaluation pipelines answer a question like:
How capable is this model?
That is useful, but production systems often need a different question:
What changed after the model update?
Consider a simple production requirement:
Return exactly three bullet points.
Do not include an introduction.
Do not include a conclusion.
Model A
- Redis is an in-memory data store.
- It supports key-value operations.
- It is commonly used for caching.
Model B
Sure! Here are three important points about Redis:
- Redis is an in-memory data store.
- It supports key-value operations.
- It is commonly used for caching.
In conclusion, Redis is widely used for high-performance applications.
Both answers contain correct information.
But Model B has violated the behavioral contract.
A benchmark may not flag this.
A production application might.
This is the class of problem llm-diff is designed to expose.
2. The Core Idea
The concept behind llm-diff is inspired by a familiar software-engineering primitive:
Codebase A
+
Codebase B
↓
git diff
↓
What changed?
For language models:
Model A
+
Model B
↓
llm-diff
↓
What changed in behavior?
Instead of replacing benchmark evaluation, llm-diff adds a behavioral regression layer between model development and deployment.
3. What llm-diff Measures
The current implementation focuses on three behavioral dimensions.
3.1 Instruction Fidelity
The first question is:
Does the model follow explicit instructions?
A probe can specify constraints such as:
Return exactly 3 bullet points.
Do not use Markdown headings.
Do not include an introduction.
The model response is then evaluated against those requirements.
For example:
Instruction Fidelity
Model A Model B Δ
Fidelity score 1.00 0.50 -0.50
A change from 1.00 to 0.50 becomes an explicit regression signal rather than something discovered after deployment.
3.2 Verbosity Profile
Model updates can also change how much the model says.
Suppose a production model previously generated:
Average response length: 42 words
After an upgrade:
Average response length: 119 words
The new model may still be correct.
But it can also introduce:
- higher inference cost
- increased latency
- larger context usage
- unnecessary explanation
- formatting changes
llm-diff tracks signals such as:
- total word count
- preamble/filler language
- Markdown header usage
Example:
Verbosity Profile
Model A Model B Δ
Total words 42 119 +77
Preamble words 0 9 +9
Markdown headers 0 3 +3
This makes behavioral drift visible.
4. Reasoning Consistency
A second important property is stability under controlled prompt variation.
Consider:
Prompt A
All researchers are engineers.
Some engineers are founders.
Can we conclude that some researchers are founders?
and an isomorphic version:
Prompt B
All astronomers are pilots.
Some pilots are musicians.
Can we conclude that some astronomers are musicians?
The terminology changes, but the underlying logical structure remains the same.
A robust reasoning system should ideally preserve the same logical verdict.
llm-diff uses this kind of controlled variation to measure whether reasoning behavior remains consistent rather than relying on one fixed prompt.
5. A Behavioral Diff
The result is intended to look familiar to an engineer.
A simplified example:
Baseline: model-a
Candidate: model-b
╭──────────────────────┬──────────────┬──────────────┬────────╮
│ Dimension │ Model A │ Model B │ Δ │
├──────────────────────┼──────────────┼──────────────┼────────┤
│ Instruction Fidelity │ 1.00 │ 0.50 │ -0.50 │
│ Total Words │ 42 │ 119 │ +77 │
│ Preamble Words │ 0 │ 9 │ +9 │
│ Markdown Headers │ 0 │ 3 │ +3 │
│ Reasoning Stability │ 1.00 │ 0.80 │ -0.20 │
╰──────────────────────┴──────────────┴──────────────┴────────╯
The important information is not simply the absolute score.
It is the delta.
That is what makes the tool useful for regression testing.
6. From Evaluation to CI/CD
The larger goal is to make behavioral evaluation part of the model-release process.
A conventional workflow might look like:
New model
↓
Run benchmark
↓
Score improved
↓
Deploy
A regression-aware workflow becomes:
New model
↓
Benchmark evaluation
↓
Behavioral regression tests
↓
Compare against production baseline
↓
Regression detected?
↙ ↘
YES NO
↓ ↓
BLOCK DEPLOY
llm-diff supports machine-readable JSON output and CI-oriented exit codes so its results can participate in automated release workflows.
A simplified CI step:
- name: LLM behavioral regression test
run: |
llm-diff \
openai/model-a \
openai/model-b \
--json > report.json
The report can then be stored as a CI artifact or evaluated against project-specific thresholds.
7. Why This Matters for MLOps
Modern LLM systems are increasingly dynamic.
Teams routinely:
- upgrade model versions
- switch providers
- fine-tune checkpoints
- merge adapters
- quantize models
- optimize inference
- change prompting strategies
- move between local and hosted deployments
Every one of those changes can introduce behavioral drift.
This is especially important when an LLM sits inside a larger system:
User
↓
Application
↓
Prompt
↓
LLM
↓
Structured output
↓
Parser / tool / database / workflow
A small behavioral change in the model can therefore propagate into a much larger system failure.
For this reason, model evaluation should not exist only at the research stage.
It should become part of the software delivery lifecycle.
8. Benchmarking vs Regression Testing
llm-diff is not intended to replace established benchmarks.
Benchmarks answer questions such as:
How capable is the model?
Regression testing answers:
What changed after the update?
Application tests answer:
Does the actual product still work?
These are different layers.
MODEL
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
Capability Behavioral Application
Benchmarks Regression Tests
│ │ │
└───────────┼───────────┘
▼
Release Gate
A production-grade evaluation stack should ideally use all three.
9. Why llm-diff Is Lightweight
The project intentionally starts small.
Rather than requiring a large evaluation infrastructure every time a developer changes a model, llm-diff is designed to function as a fast behavioral smoke test.
The goal is:
Model update
↓
Run probes
↓
Detect behavioral drift
↓
Decide whether deeper evaluation is necessary
This makes the tool complementary to large benchmark suites rather than competitive with them.
10. Backend Agnostic by Design
A regression framework should not be tightly coupled to a single provider.
llm-diff is designed to work across local and OpenAI-compatible inference environments.
This makes comparisons such as:
Local vs Local
Cloud vs Cloud
Local vs Cloud
Base vs Fine-tuned
Old checkpoint vs New checkpoint
Production vs Candidate
possible using the same evaluation concept.
The goal is to compare behavior, not infrastructure.
11. Reproducibility
Behavioral comparison is only useful when the evaluation itself is controlled.
The project therefore emphasizes deterministic evaluation settings where supported by the backend.
The development workflow also includes offline-compatible testing using mock model backends, allowing the evaluation layer itself to be tested without depending on external inference services.
This distinction is important:
The regression framework should be predictable even when the model behind it is not.
12. Installation
Install from PyPI:
pip install pluto-llm-diff
Example:
llm-diff \
ollama/qwen2.5:0.5b \
ollama/llama3.2:1b
Generate machine-readable JSON:
llm-diff \
ollama/qwen2.5:0.5b \
openai/gpt-4o-mini \
--json > diff.json
Inspect model responses:
llm-diff \
ollama/qwen2.5:0.5b \
ollama/llama3.2:1b \
--show-responses
13. What llm-diff Is Not
llm-diff is not a replacement for:
- MMLU
- HumanEval
- GPQA
- domain-specific benchmarks
- human evaluation
- application-level tests
- safety evaluation
Instead, it occupies a different layer:
behavioral regression testing between model versions.
That distinction is fundamental to the project.
14. The Research Question Behind the Tool
The project began as an engineering problem, but it leads to a broader research question:
Can a model improve on conventional benchmarks while becoming worse for the behavior an application actually depends on?
Consider a hypothetical model release:
Model A Model B
MMLU 74.1 76.3 ↑
Coding benchmark 61.2 64.0 ↑
Instruction fidelity 0.96 0.71 ↓
Average response length 48 131 ↑
Reasoning consistency 0.94 0.81 ↓
If we examine only benchmark scores:
Model B looks better.
If we examine the complete behavioral profile:
the upgrade becomes much more complicated.
This is the gap llm-diff tries to make visible.
15. Future Direction
The current system is intentionally focused.
Future work includes expanding the behavioral test suite toward:
- hallucination regression
- refusal behavior
- domain retention
- sycophancy
- custom YAML probe packs
- N-way model comparison
- threshold-based release gates
- richer CI/CD integrations
- Hugging Face-native evaluation workflows
The long-term objective is simple:
Make behavioral regression testing as normal for model releases as unit testing is for software releases.
16. Open Source
llm-diff is developed as part of Pluto AI Research Lab, an independent open-source research initiative focused on efficient AI, model distillation, reasoning, multimodal systems, evaluation, and edge deployment.
The project is open source and available for researchers and engineers to inspect, extend, and integrate into their own workflows.
Project Links
- 💻 GitHub: https://github.com/Pluto-AI-Labs/llm-diff
- 🌐 Website: https://pluto-llm-diff.vercel.app/
- 📦 PyPI: https://pypi.org/project/pluto-llm-diff/
- 📄 Whitepaper: https://github.com/Pluto-AI-Labs/llm-diff/blob/main/WHITEPAPER.md
- 🪐 Pluto AI Labs: https://huggingface.co/Pluto-AI-Labs
Conclusion
The LLM ecosystem has become very good at answering:
Which model scores higher?
Production systems increasingly need another question:
What changed?
A benchmark can tell us that a model improved on a particular task.
A behavioral regression test can tell us whether the model still behaves the way our software expects.
llm-diff is an attempt to make that second question measurable, automatable, and suitable for modern AI engineering workflows.
Benchmarks measure capability.
llm-diffmeasures behavioral change.
About the Author
Siddharth N.R. is the Founder and Lead AI Engineer of Pluto AI Research Lab, working on efficient LLMs, multimodal models, model distillation, evaluation infrastructure, and edge AI.
Intelligence isn't about scale. It's about precision.
