--- license: apache-2.0 library_name: transformers base_model: Qwen/Qwen2-0.5B-Instruct tags: - generated_from_trainer - trl - sft - sql - text model_name: Qwen2-0.5B-Instruct-SQL-generator datasets: - gretelai/synthetic_text_to_sql language: - en metrics: - bleu - chrf - rouge --- # Model Card for Qwen2-0.5B-Instruct-SQL-generator This model is a fine-tuned version of [Qwen/Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct). It has been trained using [TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl) for SQL generation tasks. ## Quick Start ```python from transformers import pipeline question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?" generator = pipeline("text-generation", model="onkolahmet/Qwen2-0.5B-Instruct-SQL-generator", device="cuda") output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0] print(output["generated_text"]) ``` ## Training Procedure This model was trained with Supervised Fine-Tuning (SFT) using the [gretelai/synthetic_text_to_sql](https://huggingface.co/datasets/gretelai/synthetic_text_to_sql) dataset. The goal was to fine-tune the model to better translate natural language queries into SQL statements. ### Framework Versions - TRL: 0.12.2 - Transformers: 4.46.3 - PyTorch: 2.6.0 - Datasets: 3.4.1 - Tokenizers: 0.20.3 ## Evaluation Results The model was evaluated using standard text generation metrics (BLEU, ROUGE-L F1, CHRF) in both zero-shot and few-shot prompting scenarios. ### 🔹 Zero-shot Prompting (on `gretelai/synthetic_text_to_sql/test`) **After Post-processing:** - **BLEU Score:** 0.5195 - **ROUGE-L F1:** 0.7031 - **CHRF Score:** 70.0409 **Before Post-processing:** - **BLEU Score:** 0.1452 - **ROUGE-L F1:** 0.3009 - **CHRF Score:** 47.8182 **SQL-Specific Metrics:** - **Exact Match (case insensitive):** 0.1600 - **Normalized Exact Match:** 0.1500 - **Average Component Match:** 0.4528 - **Average Entity Match:** 0.8807 **Query Quality Distribution:** - **High Quality (≥80% component match):** 18 (18.0%) - **Medium Quality (50-79% component match):** 28 (28.0%) - **Low Quality (<50% component match):** 54 (54.0%) --- ### 🔹 Few-shot Prompting (on `gretelai/synthetic_text_to_sql/test`) **After Post-processing:** - **BLEU Score:** 0.2680 - **ROUGE-L F1:** 0.4975 - **CHRF Score:** 57.1704 **Before Post-processing:** - **BLEU Score:** 0.1272 - **ROUGE-L F1:** 0.2816 - **CHRF Score:** 46.1643 **SQL-Specific Metrics:** - **Exact Match (case insensitive):** 0.0000 - **Normalized Exact Match:** 0.0000 - **Average Component Match:** 0.2140 - **Average Entity Match:** 0.8067 **Query Quality Distribution:** - **High Quality (≥80% component match):** 4 (4.0%) - **Medium Quality (50-79% component match):** 17 (17.0%) - **Low Quality (<50% component match):** 79 (79.0%) ## Citations Cite TRL as: ```bibtex @misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}} } ```