ArtificialGuyBR

Home / Blog / Fixing things

Llama3 8B SQL Create Context: Text-to-SQL with Schema Grounding

A fine-tuned Llama 3 8B model that generates accurate SQL queries from natural language using CREATE TABLE statements as context.

1 sources cited Fixing things

On this page

  1. How It Works
  2. Training Details
  3. When to Use This Model
  4. Setup and Usage
  5. Limitations
  6. FAQ
  7. Sources

Llama3 8B SQL Create Context: Text-to-SQL with Schema Grounding

Converting natural language questions into SQL queries is one of the most practical applications of fine-tuned LLMs. Llama3 8B SQL Create Context takes a focused approach: it generates SQL by grounding every output in the actual table schema, preventing the hallucination problems that plague most text-to-SQL systems.

How It Works

The model accepts two inputs: a natural language question (e.g., "What is the total revenue by region?") and a CREATE TABLE statement defining the relevant schema. It then generates a SQL query that answers the question using only the tables and columns described in the schema.

This approach differs from open-ended text-to-SQL models that hallucinate table or column names. By requiring a CREATE TABLE statement as context, the model always has the schema available and is more likely to produce syntactically and semantically correct queries.

Example interaction:

Input:
CREATE TABLE sales (
  id INT PRIMARY KEY,
  region VARCHAR(50),
  amount DECIMAL(10,2),
  sale_date DATE
);

Question: What are the top 3 regions by total sales amount?

Output:
SELECT region, SUM(amount) AS total_amount
FROM sales
GROUP BY region
ORDER BY total_amount DESC
LIMIT 3;

Training Details

The model was fine-tuned using Axolotl on a combination of the WikiSQL and Spider datasets, totaling 78,577 training examples. Key hyperparameters: Parameter | Value

|---|---

Base Model | NousResearch/Meta-Llama-3-8B Learning Rate | 2e-5 Scheduler | Cosine with 100 warmup steps Optimizer | Paged AdamW 8-bit Epochs | 3 Sequence Length | 8192 Batch Size | 1 (gradient accumulation: 8) Attention | Flash Attention Training loss dropped from 0.7175 at step 1 to 0.0106 by epoch 2.5, with validation loss stabilizing at 0.0201.

When to Use This Model

The model works best when:

It may struggle with:

Setup and Usage

Load the model with HuggingFace Transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "artificialguybr/llama3-8b-sql-create-context"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

prompt = """CREATE TABLE employees (
  id INT PRIMARY KEY,
  name VARCHAR(100),
  department VARCHAR(50),
  salary DECIMAL(10,2)
);

Question: How many employees are in each department?"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For inference at scale, the model is compatible with text-generation-inference (TGI) and can be deployed via HuggingFace endpoints.

Limitations

The model relies on accurate CREATE TABLE statements — incomplete or incorrect schemas will produce incorrect queries. It is not designed for analytical queries requiring external knowledge, common-sense reasoning, or queries across schemas not provided in the prompt.

FAQ

Q: Does this model work with any SQL dialect? A: It was trained on standard SQL from WikiSQL and Spider. It may not handle dialect-specific syntax (e.g., PostgreSQL-specific functions, T-SQL) without fine-tuning.

Q: What's the difference between this and a generic text-to-SQL model? A: This model specifically requires CREATE TABLE context, which anchors generation to the real schema and prevents hallucination. Generic models may invent non-existent tables or columns.

Q: Can I fine-tune it further for my own database? A: Yes. The Axolotl config is included in the model card, and additional training on your schema-question pairs should improve domain-specific accuracy.

Q: What's the evaluation loss? A: 0.0201 on the held-out validation set (5% split of the training data).

Sources