File size: 2,898 Bytes
db7e5b2 921c7c2 c1e87c2 13c5628 db7e5b2 c1e87c2 db7e5b2 8d65c2d c1e87c2 dcd50ca db7e5b2 dcd50ca db7e5b2 9e740f7 dcd50ca 9e740f7 c1e87c2 a66de90 db7e5b2 a66de90 c1e87c2 a66de90 9066ac6 a66de90 9e740f7 9066ac6 dcd50ca a66de90 9066ac6 a66de90 9066ac6 c1e87c2 db7e5b2 c1e87c2 db7e5b2 c1e87c2 db7e5b2 c1e87c2 db7e5b2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
---
base_model: google/gemma-2b
library_name: peft
license: bsl-1.0
tags:
- code
datasets:
- b-mc2/sql-create-context
language:
- en
pipeline_tag: text2text-generation
---
# Model Card for Model ID
### Model Description
This model is quantized in 8-bit and trained with question and answer pairs for text-to-SQL tasks using the LoRA PEFT method. It serves as a foundation model for further development in Text-to-SQL Retrieval-Augmented Generation (RAG) applications.
- **Developed by:** Lei-bw
- **Model type:** Causal Language Model
- **Language(s) (NLP):** English
- **License:** bsl-1.0
- **Finetuned from model:** google/gemma-2b
- **Device to be used:** NVIDIA GeForce RTX 3080 Ti (12288MiB)
## How to Get Started with the Model
1. Apply the access to model google/gemma-2b
You may submit the approval from [google/gemma-2b](https://huggingface.co/google/gemma-2b)
2. Login-in
If you are using cli, please use below command to login your huggingface account with your access token key:
```bash
huggingface-cli login
```
OR
If you are using notebook, please use below command to login your huggingface account with your access token key:
```python
from huggingface_hub import login
login()
```
3. Install the required library
```python
peft==0.12.0
transformers==4.44.2
```
4. Get access to the `Lei-bw/text-to-sql-fm` model
```python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
config = PeftConfig.from_pretrained("Lei-bw/text-to-sql-fm")
base_model = AutoModelForCausalLM.from_pretrained("google/gemma-2b")
model = PeftModel.from_pretrained(base_model, "Lei-bw/text-to-sql-fm")
tokenizer = AutoTokenizer.from_pretrained("Lei-bw/text-to-sql-fm")
```
5. Example to use this model:
```python
import torch
# Example usage
eval_prompt = """
How many seniors in the departments are younger than 28?
"""
model_input = tokenizer(eval_prompt, return_tensors="pt")
model.eval()
with torch.no_grad():
print(tokenizer.decode(model.generate(**model_input, max_new_tokens=300)[0], skip_special_tokens=True))
```
## Training Details
### Training Data
The model was trained on the b-mc2/sql-create-context dataset, which contains question and answer pairs for SQL generation tasks.
#### Training Hyperparameters
• Training regime: bf16 mixed precision
• Batch size: 16
• Gradient accumulation steps: 4
• Warmup steps: 50
• Number of epochs: 2
• Learning rate: 2e-4
• Weight decay: 0.01
• Optimizer: AdamW
• Learning rate scheduler: Linear
#### Hardware
- **Hardware Type:** NVIDIA A100
- **GPU RAM:** 40 GB
### Framework versions
- PEFT 0.12.0 |