|
|
--- |
|
|
library_name: transformers |
|
|
base_model: meta-llama/Meta-Llama-3-8B-Instruct |
|
|
license: llama3 |
|
|
language: |
|
|
- pt |
|
|
tags: |
|
|
- code |
|
|
- sql |
|
|
- finetuned |
|
|
- portugues-BR |
|
|
co2_eq_emissions: |
|
|
emissions: 1450 |
|
|
source: "Lacoste, Alexandre, et al. “Quantifying the Carbon Emissions of Machine Learning.” ArXiv (Cornell University), 21 Oct. 2019, https://doi.org/10.48550/arxiv.1910.09700." |
|
|
training_type: "fine-tuning" |
|
|
geographical_location: "Council Bluffs, Iowa, USA." |
|
|
hardware_used: "1 A100 40GB GPU" |
|
|
--- |
|
|
# Lloro SQL |
|
|
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/> |
|
|
|
|
|
Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct, that was trained on GretelAI public datasets. The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM. |
|
|
|
|
|
## Model description |
|
|
|
|
|
Model type: A 7B parameter fine-tuned on GretelAI public datasets. |
|
|
|
|
|
Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well |
|
|
|
|
|
Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct |
|
|
|
|
|
## What is Lloro's intended use(s)? |
|
|
|
|
|
Lloro is built for Text2SQL in Portuguese contexts . |
|
|
|
|
|
Input : Text |
|
|
|
|
|
Output : Text (Code) |
|
|
|
|
|
## Usage |
|
|
|
|
|
Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html)) |
|
|
|
|
|
```python |
|
|
from openai import OpenAI |
|
|
client = OpenAI( |
|
|
api_key="EMPTY", |
|
|
base_url="http://localhost:8000/v1", |
|
|
) |
|
|
def generate_responses(instruction, client=client): |
|
|
|
|
|
chat_response = client.chat.completions.create( |
|
|
model=<model>, |
|
|
messages=[ |
|
|
{"role": "system", "content": "Você escreve a instrução SQL que responde às perguntas feitas. Você NÃO FORNECE NENHUM COMENTÁRIO OU EXPLICAÇÃO sobre o que o código faz, apenas a instrução SQL terminando em ponto e vírgula. Você utiliza todos os comandos disponíveis na especificação SQL, como: [SELECT, WHERE, ORDER, LIMIT, CAST, AS, JOIN]."}, |
|
|
{"role": "user", "content": instruction}, |
|
|
] |
|
|
) |
|
|
|
|
|
return chat_response.choices[0].message.content |
|
|
|
|
|
output = generate_responses(user_prompt) |
|
|
|
|
|
``` |
|
|
|
|
|
## Params |
|
|
|
|
|
Training Parameters |
|
|
| Params | Training Data | Examples | Tokens | LR | |
|
|
|----------------------------------|-------------------------------------------|---------------------------------|------------|--------| |
|
|
| 8B | GretelAI public datasets + Synthetic Data | 102970 | 18.654.222 | 2e-4 | |
|
|
|
|
|
## Model Sources |
|
|
|
|
|
GretelAI: <https://huggingface.co/datasets/gretelai/synthetic_text_to_sql> |
|
|
|
|
|
## Performance |
|
|
|
|
|
### Test Dataset |
|
|
|
|
|
| Model | LLM as Judge | Code Bleu Score | Rouge-L | CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 | |
|
|
|----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------| |
|
|
| Llama 3 8B | 65.48% | 0.4583 | 0.6361 | 0.8815 | 0.8871 | 0.8835 | 0.8862 | |
|
|
| Lloro - SQL | 71.33% | 0.6512 | 0.7965 | 0.9458 | 0.9469 | 0.9459 | 0.9466 | |
|
|
| GPT - 3.5 Turbo| 67.52% | 0.6232 | 0.9967 | 0.9151 | 0.9152 | 0.9142 | 0.9175 | |
|
|
|
|
|
### Database Benchmark |
|
|
|
|
|
| Model | Score | |
|
|
|----------------|--------------| |
|
|
| Llama 3 - Base | 35.55% | |
|
|
| Lloro - SQL | 49.48% | |
|
|
| GPT - 3.5 Turbo| 46.15% | |
|
|
|
|
|
### Translated BIRD Benchmark - https://bird-bench.github.io/ |
|
|
|
|
|
| Model | Score | |
|
|
|----------------|--------------| |
|
|
| Llama 3 - Base | 33.87% | |
|
|
| Lloro - SQL | 47.14% | |
|
|
| GPT - 3.5 Turbo| 42.14% | |
|
|
|
|
|
## Training Infos |
|
|
|
|
|
The following hyperparameters were used during training: |
|
|
|
|
|
| Parameter | Value | |
|
|
|---------------------------|----------------------| |
|
|
| learning_rate | 2e-4 | |
|
|
| weight_decay | 0.001 | |
|
|
| train_batch_size | 16 | |
|
|
| eval_batch_size | 8 | |
|
|
| seed | 42 | |
|
|
| optimizer | Adam - adamw_8bit | |
|
|
| lr_scheduler_type | cosine | |
|
|
| num_epochs | 4.0 | |
|
|
|
|
|
## QLoRA hyperparameters |
|
|
|
|
|
The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training: |
|
|
|
|
|
| Parameter | Value | |
|
|
|-----------------|---------| |
|
|
| lora_r | 64 | |
|
|
| lora_alpha | 128 | |
|
|
| lora_dropout | 0 | |
|
|
|
|
|
## Experiments |
|
|
|
|
|
| Model | Epochs | Overfitting | Final Epochs | Training Hours | CO2 Emission (Kg) | |
|
|
|-----------------------|--------|-------------|--------------|-----------------|-------------------| |
|
|
| Llama 3 8B Instruct | 5 | Yes | 4 | 10.16 | 1.45 | |
|
|
|
|
|
## Framework versions |
|
|
|
|
|
| Library | Version | |
|
|
|---------------|-----------| |
|
|
| accelerate | 0.21.0 | |
|
|
| bitsandbytes | 0.42.0 | |
|
|
| Datasets | 2.14.3 | |
|
|
| peft | 0.4.0 | |
|
|
| Pytorch | 2.0.1 | |
|
|
| safetensors | 0.4.1 | |
|
|
| scikit-image | 0.22.0 | |
|
|
| scikit-learn | 1.3.2 | |
|
|
| Tokenizers | 0.14.1 | |
|
|
| Transformers | 4.37.2 | |
|
|
| trl | 0.4.7 | |
