Llama3 8B Redmond Code290k: Code Generation via Fine-Tuning

TL;DR

• Fine-tuned from NousResearch/Meta-Llama-3-8B on 290k code-focused conversations
• Supports Python, Java, JavaScript, Go, C++, Rust, Ruby, SQL, and more
• Trained with Axolotl, 8-bit Adam, flash attention, 8K context

What This Model Does

Llama3 8B Redmond Code290k takes LLaMA 3 8B and specializes it for code generation and explanation. Where LLaMA 3 is a general-purpose model, this variant sharpens code output quality by training on nearly 300k instruction-response pairs in the Vicuna/ShareGPT format.

Training Details

Trained with Axolotl v0.4.0:

• Dataset: ajibawa-2023/Code-290k-ShareGPT (~290,000 conversations)
• Learning rate: 2e-05 with cosine scheduling, 100 warmup steps
• Epochs: 3 (2 per hyperparameters documentation)
• Context length: 8192 tokens
• Optimizer: 8-bit Adam with gradient checkpointing and flash attention
• Hardware: GPU support courtesy of Redmond.ai

How to Use

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="artificialguybr/llama3-8b-redmond-code290k",
    max_new_tokens=512,
)

prompt = "Write a Python function that reverses a linked list."
result = generator(prompt)
print(result[0]["generated_text"])

The model is also endpoints compatible and works with text-generation-inference deployments.

Supported Languages

Python, Java, JavaScript, Go, C++, Rust, Ruby, SQL, MySQL, R, Julia, Haskell, and others included in the training dataset.

Limitations

• Reduced performance on rare or niche programming languages
• May not generalize to unseen coding styles or conventions
• Struggles with extremely complex code or edge cases
• Limited explanations for highly abstract or theoretical concepts

Sources

• Hugging Face Model Page — model card, training config, and Axolotl yaml
• Base Model: NousResearch/Meta-Llama-3-8B — upstream LLaMA 3 8B
• Dataset: ajibawa-2023/Code-290k-ShareGPT — training data