Llama3 8B SQL Create Context: Text-to-SQL with Schema Grounding
A fine-tuned Llama 3 8B model that generates accurate SQL queries from natural language using CREATE TABLE statements as context.
Converting natural language questions into SQL queries is one of the most practical applications of fine-tuned LLMs. Llama3 8B SQL Create Context takes a focused approach: it generates SQL by grounding every output in the actual table schema, preventing the hallucination problems that plague most text-to-SQL systems.
- Fine-tuned from NousResearch/Meta-Llama-3-8B on 78,577 SQL examples
- Uses CREATE TABLE statements as input context to anchor generation
- Validated on combined WikiSQL and Spider datasets with 0.0201 loss
How It Works
The model accepts two inputs: a natural language question (e.g., "What is the total revenue by region?") and a CREATE TABLE statement defining the relevant schema. It then generates a SQL query that answers the question using only the tables and columns described in the schema.
This approach differs from open-ended text-to-SQL models that hallucinate table or column names. By requiring a CREATE TABLE statement as context, the model always has the schema available and is more likely to produce syntactically and semantically correct queries.
Example interaction:
Input:
CREATE TABLE sales (
id INT PRIMARY KEY,
region VARCHAR(50),
amount DECIMAL(10,2),
sale_date DATE
);
Question: What are the top 3 regions by total sales amount?
Output:
SELECT region, SUM(amount) AS total_amount
FROM sales
GROUP BY region
ORDER BY total_amount DESC
LIMIT 3;
Training Details
The model was fine-tuned using Axolotl on a combination of the WikiSQL and Spider datasets, totaling 78,577 training examples. Key hyperparameters: Parameter | Value
|---|---
Base Model | NousResearch/Meta-Llama-3-8B Learning Rate | 2e-5 Scheduler | Cosine with 100 warmup steps Optimizer | Paged AdamW 8-bit Epochs | 3 Sequence Length | 8192 Batch Size | 1 (gradient accumulation: 8) Attention | Flash Attention Training loss dropped from 0.7175 at step 1 to 0.0106 by epoch 2.5, with validation loss stabilizing at 0.0201.
When to Use This Model
The model works best when:
- The table schema is known and provided as a CREATE TABLE statement
- Queries are straightforward (SELECT, WHERE, GROUP BY, ORDER BY, LIMIT)
- You need to prevent hallucination of table/column names
It may struggle with:
- Complex multi-join queries
- Subqueries and nested logic
- Ambiguous or open-ended questions
- Unseen SQL dialects
Setup and Usage
Load the model with HuggingFace Transformers:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "artificialguybr/llama3-8b-sql-create-context"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
prompt = """CREATE TABLE employees (
id INT PRIMARY KEY,
name VARCHAR(100),
department VARCHAR(50),
salary DECIMAL(10,2)
);
Question: How many employees are in each department?"""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For inference at scale, the model is compatible with text-generation-inference (TGI) and can be deployed via HuggingFace endpoints.
Limitations
The model relies on accurate CREATE TABLE statements — incomplete or incorrect schemas will produce incorrect queries. It is not designed for analytical queries requiring external knowledge, common-sense reasoning, or queries across schemas not provided in the prompt.
FAQ
Q: Does this model work with any SQL dialect? A: It was trained on standard SQL from WikiSQL and Spider. It may not handle dialect-specific syntax (e.g., PostgreSQL-specific functions, T-SQL) without fine-tuning.
Q: What's the difference between this and a generic text-to-SQL model? A: This model specifically requires CREATE TABLE context, which anchors generation to the real schema and prevents hallucination. Generic models may invent non-existent tables or columns.
Q: Can I fine-tune it further for my own database? A: Yes. The Axolotl config is included in the model card, and additional training on your schema-question pairs should improve domain-specific accuracy.
Q: What's the evaluation loss? A: 0.0201 on the held-out validation set (5% split of the training data).
Sources
- HuggingFace Model Card — Full training details, config, and evaluation results
- b-mc2/sql-create-context Dataset — Training dataset combining WikiSQL and Spider