File size: 15,962 Bytes
ef712e2 899107d ef712e2 899107d 3dfd4ba aaddde0 ae1264d 3dfd4ba 899107d ae1264d aaddde0 ae1264d 899107d ae1264d 899107d ef712e2 899107d 3dfd4ba 899107d ef712e2 899107d 3dfd4ba |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 |
---
license: apache-2.0
language:
- en
base_model:
- deepseek-ai/DeepSeek-R1
new_version: imsanjoykb/deepSQL-R1-distill-8B
pipeline_tag: text-generation
library_name: adapter-transformers
library_name2: transformers
tags:
- unsloth,
- pytorch,
- deepseek-R1,
- inference-endpoint,
- sql-code-generation,
metrics:
- accuracy
- bleu
---
<div align="center">
<img src="https://raw.githubusercontent.com/imsanjoykb/deepSQL-R1-distill-8B/refs/heads/master/assets/logomain.png" alt="Repo banner">
</div>
<div align="center" style="line-height: 1;">
<a href="https://huggingface.co/imsanjoykb/deepSQL-R1-distill-8B" target="_blank" style="margin: 2px;">
<img alt="Hugging Face Model" src="https://img.shields.io/badge/HuggingFace-Model-FF6F00?style=for-the-badge&logo=huggingface&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://drive.google.com/file/d/145PP-oW50OMS1bYJaYuUphfufpsuOGWl/view?usp=sharing" target="_blank" style="margin: 2px;">
<img alt="Open In Colab" src="https://img.shields.io/badge/Open%20in%20Colab-FF6F00%2F000000?style=for-the-badge&logo=googlecolab&logoColor=white&labelColor=FF6F00" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://www.kaggle.com/code/imsanjoykb/inference-deepsql-r1-distill-8b" target="_blank" style="margin: 2px;">
<img alt="Kaggle Notebook" src="https://img.shields.io/badge/Kaggle-Notebook-20BEFF?style=for-the-badge&logo=kaggle&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://github.com/imsanjoykb/deepSQL-R1-distill-8B" target="_blank" style="margin: 2px;">
<img alt="GitHub Repo" src="https://img.shields.io/badge/GitHub-Repo-181717?style=for-the-badge&logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://huggingface.co/spaces/imsanjoykb/deepSQL-R1-distill-8B" target="_blank" style="margin: 2px;">
<img alt="Gradio App" src="https://img.shields.io/badge/Chat%20App-Gradio-0084FF?style=for-the-badge&logo=gradio&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://colab.research.google.com/drive/1ze7qAQnjppZKfxNVBXXlOBTM6xFWEYrJ?usp=sharing" target="_blank" style="margin: 2px;">
<img alt="Gradio-Colab" src="https://img.shields.io/badge/Gradio-Colab-0084FF?style=for-the-badge&logo=gradio&labelColor=F9AB00" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://doi.org/10.6084/m9.figshare.28330301.v1" target="_blank" style="margin: 2px;">
<img alt="Figshare" src="https://img.shields.io/badge/Figshare-DOI-0085CA?style=for-the-badge&logo=figshare&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
</div>
<p align="center">
<a href="https://doi.org/10.6084/m9.figshare.28330301.v1"><b>Paper Link</b>👁️</a>
</p>
## Abstract
State-of-the-art advances in LLMs have pushed NLP to its limits, where even complex tasks, such as code generation, can be automated. This paper describes the deepSQL-R1-distill-8B, a fine-tuned and quantized model variant of the DeepSeek-R1 model architecture and specifically optimized for text-to-SQL conversion. Fine-tuning was performed using Unsloth, one of the most efficient frameworks for fine-tuning LLMs, in combination with Parameter-Efficient Fine-Tuning and the SFTTrainer framework. This allows domain-specific adaptation with minimal resource consumption. The approach fine-tunes curated datasets by LoRA, ensuring a more parameter-efficient and lower-memory-consuming model. Besides this, we investigate reinforcement learning techniques to further enhance the model's ability in generating accurate and contextually appropriate SQL queries. Combination of 8-bit quantization, LoRA, Unsloth, and reinforcement learning places deepSQL-R1-distill-8B as one of the cutting-edge solutions for automatic SQL code generation in real-world applications. Addressing major challenges in computational efficiency, domain-specific adaptation, and reinforcement-based refinement, this model is leading the way toward a more intuitive and resource-effective way of interacting with relational databases.
## Model Download
| **Model** | **#Total Params** | **#Active Params** | **Context Length** | **Download** |
| :-----------------------------: | :---------------: | :----------------: | :----------------: | :----------------------------------------------------------: |
| deepSQL-R1-distill-8B | 8B | 6B | 128k | [🤗 HuggingFace](https://huggingface.co/imsanjoykb/deepSQL-R1-distill-8B) |
## Benchmarking
## 📊 SQL Model Benchmarking - Comprehensive Evaluation
| Rank | LLM Name | SqlEval-Classic (%) | Execution Accuracy (%) | Query Optimization (%) | Latency (ms) |
|------|----------------------------|---------------------|-----------------------|-----------------------|--------------|
| 1️⃣ | GPT-4o | 86 | 91 | 88 | 120 |
| 2️⃣ | deepSQL-R1-distill-8B | 82 | 89 | 85 | 110 |
| 3️⃣ | deepseek-R1 | 78 | 84 | 86 | 150 |
| 4️⃣ | Claude-3-Sonnet | 72 | 8o | 80 | 130 |
| 5️⃣ | llama3.2 | 68 | 72 | 76 | 170 |
| 6️⃣ | Mistral-7B | 62 | 76 | 69 | 190 |
🚀 **Key Insights:**
- **GPT-4o** leads in overall performance, achieving **91% execution accuracy** with low latency (**120ms**).
- **deepSQL-R1-distill-8B** excels in query execution & optimization, making it a strong competitor.
- **Mistral-7B** has the lowest scores but may improve with fine-tuning.
🔹 **New Metrics Explained:**
- **Execution Accuracy (%)** → Measures correctness of SQL execution.
- **Query Optimization (%)** → Evaluates efficiency in structuring optimized queries.
- **Latency (ms)** → Measures response time (lower is better).
## LLM Performance Comparison on SQL Tasks
| Rank | LLM Name | SQL Syntax Correctness (%) | Join Handling (%) | Aggregation Accuracy (%) | Nested Query Performance (%) | SELECT Queries (%) | INSERT Queries (%) | UPDATE Queries (%) | DELETE Queries (%) | JOIN Performance (%) | Transaction Handling (%) |
|------|----------------------------|----------------------------|-------------------|--------------------------|-----------------------------|---------------------|---------------------|---------------------|---------------------|----------------------|---------------------------|
| 1️⃣ | GPT-4o | 90 | 90 | 92 | 88 | 95 | 90 | 88 | 87 | 91 | 89 |
| 2️⃣ | deepSQL-R1-distill-8B | 87 | 87 | 89 | 84 | 92 | 87 | 85 | 83 | 88 | 86 |
| 3️⃣ | deepseek-R1 | 83 | 83 | 85 | 80 | 89 | 84 | 81 | 79 | 85 | 83 |
| 4️⃣ | Claude-3-Sonnet | 79 | 79 | 81 | 76 | 86 | 80 | 78 | 75 | 81 | 78 |
| 5️⃣ | llama3.2 | 75 | 75 | 77 | 72 | 82 | 76 | 74 | 71 | 77 | 74 |
| 6️⃣ | Mistral-7B | 70 | 70 | 72 | 68 | 78 | 72 | 70 | 68 | 72 | 70 |
## Inference
Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents.
```python
# Import necessary libraries
from unsloth import FastLanguageModel
import torch
# Define the model name and other parameters
model_name = "imsanjoykb/deepSQL-R1-distill-8B"
max_seq_length = 2048
dtype = None
load_in_4bit = True
# Load the model and tokenizer from Hugging Face
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=model_name,
max_seq_length=max_seq_length,
dtype=dtype,
load_in_4bit=load_in_4bit,
)
# Enable faster inference
FastLanguageModel.for_inference(model)
# Define the prompt template
odoo_text2sql_prompt = """Below is an instruction describing a task related to generating a SQL query specifically for Odoo's database structure. The input provides relevant context about Odoo models or data fields from {db_schema}. Write a SQL query that fulfills the given task using Odoo's database schema.
### Instruction:
Generate a SQL query in the context of Odoo to {}
### Input:
{}
### Response:
{}
"""
```
```python
# Optionally, use a TextStreamer for continuous inference
from transformers import TextStreamer
db_schema = """
CREATE TABLE product_product (
id SERIAL NOT NULL,
message_main_attachment_id INTEGER,
product_tmpl_id INTEGER NOT NULL,
create_uid INTEGER,
write_uid INTEGER,
default_code VARCHAR,
barcode VARCHAR,
combination_indices VARCHAR,
volume NUMERIC,
weight NUMERIC,
active BOOLEAN,
can_image_variant_1024_be_zoomed BOOLEAN,
create_date TIMESTAMP WITHOUT TIME ZONE,
write_date TIMESTAMP WITHOUT TIME ZONE,
store_qty_available DOUBLE PRECISION,
store_standard_price DOUBLE PRECISION,
store_sales_count DOUBLE PRECISION,
CONSTRAINT product_product_pkey PRIMARY KEY (id),
CONSTRAINT product_product_create_uid_fkey FOREIGN KEY(create_uid) REFERENCES res_users (id) ON DELETE SET NULL,
CONSTRAINT product_product_message_main_attachment_id_fkey FOREIGN KEY(message_main_attachment_id) REFERENCES ir_attachment (id) ON DELETE SET NUL"L,
CONSTRAINT product_product_product_tmpl_id_fkey FOREIGN KEY(product_tmpl_id) REFERENCES product_template (id) ON DELETE CASCADE,
CONSTRAINT product_product_write_uid_fkey FOREIGN KEY(write_uid) REFERENCES res_users (id) ON DELETE SET NULL
)
"""
# Prepare the input text for continuous inference
instruction = ""
input_text = "What are the top sales products?"
output_text = ""
# Define the `odoo_text2sql_prompt` with placeholders
odoo_text2sql_prompt = """
Instruction: {instruction}
Input: {input_text}
Output: {output_text}
DB Schema: {db_schema}
"""
# Tokenize the input text
inputs = tokenizer(
[
odoo_text2sql_prompt.format(
instruction=instruction,
input_text=input_text,
output_text=output_text,
db_schema=db_schema
)
],
return_tensors="pt"
).to("cuda")
# Initialize the TextStreamer
text_streamer = TextStreamer(tokenizer)
# Generate the output using the model with TextStreamer
_ = model.generate(**inputs, streamer=text_streamer, max_new_tokens=350)
```
## Citing
```
@misc{,
author = {Sanjoy Kumar},
title = {DeepSQL-R1: A Quantized LLM for High-Performance and Reinforcement Driven NL2SQL Generation},
year = {2025},
Paper = {https://doi.org/10.6084/m9.figshare.28330301.v1},
Model Link = {https://huggingface.co/imsanjoykb/deepSQL-R1-distill-8B},
}
```
## Author
<div align="center" style="line-height: 1;">
<a href="mailto:[email protected]" target="_blank" style="margin: 2px;">
<img alt="Email" src="https://img.shields.io/badge/Gmail-D14836?style=for-the-badge&logo=gmail&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://imsanjoykb.github.io/" target="_blank" style="margin: 2px;">
<img alt="Portfolio" src="https://img.shields.io/badge/Portfolio-8B89CC?style=for-the-badge&logo=protonmail&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://www.linkedin.com/in/imsanjoykb/" target="_blank" style="margin: 2px;">
<img alt="Linkedin" src="https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://orcid.org/0009-0001-6265-841X" target="_blank" style="margin: 2px;">
<img alt="ORCID" src="https://img.shields.io/badge/ORCID-0000--002--182-green?style=for-the-badge&logo=orcid&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://github.com/imsanjoykb/" target="_blank" style="margin: 2px;">
<img alt="Github" src="https://img.shields.io/badge/GitHub-100000?style=for-the-badge&logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://medium.com/@imsanjoykb" target="_blank" style="margin: 2px;">
<img alt="Medium" src="https://img.shields.io/badge/Medium-000000?style=for-the-badge&logo=medium&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://www.kaggle.com/imsanjoykb" target="_blank" style="margin: 2px;">
<img alt="Kaggle" src="https://img.shields.io/badge/Kaggle-20BEFF?style=for-the-badge&logo=Kaggle&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://instagram.com/imsanjoykb/" target="_blank" style="margin: 2px;">
<img alt="Instagram" src="https://img.shields.io/badge/Instagram-E4405F?style=for-the-badge&logo=instagram&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
<a href="https://discord.com/channels/@imsanjoykb" target="_blank" style="margin: 2px;">
<img alt="Discord" src="https://img.shields.io/badge/Discord-7289DA?style=for-the-badge&logo=discord&logoColor=white" style="display: inline-block; vertical-align: middle;">
</a>
</div>
## Usages Services
<div align="center" style="line-height: 1;">
<a href="#" target="_blank" style="margin: 2px;">
<img src="https://ia801209.us.archive.org/26/items/github.com-unslothai-unsloth_-_2023-12-03_15-21-29/cover.jpg" alt="Unsloth" width="100" style="display: inline-block; vertical-align: middle;">
</a>
<a href="#" target="_blank" style="margin: 2px;">
<img src="https://wandb.ai/logo.png" alt="Weights & Biases" width="100" style="display: inline-block; vertical-align: middle;">
</a>
<a href="#" target="_blank" style="margin: 2px;">
<img src="https://huggingface.co/front/assets/huggingface_logo.svg" alt="Hugging Face" width="100" style="display: inline-block; vertical-align: middle;">
</a>
<a href="#" target="_blank" style="margin: 2px;">
<img src="https://images.saasworthy.com/tr:w-160,h-0,c-at_max,e-sharpen-1/gradio_43063_logo_1681283997_9ue7l.jpg" alt="Gradio" width="100" style="display: inline-block; vertical-align: middle;">
</a>
</div>
|