ArtificialGuyBR

Home / Blog / Fixing things

Llama3 8B Redmond Code290k: Fine-Tuned Code Generation

Fine-tuned LLaMA 3 8B model for code generation across Python, Java, Go, and more

By ArtificialGuyBR Published Updated 1 sources cited Fixing things

TL;DR

  • Fine-tuned from NousResearch/Meta-Llama-3-8B on 290k code-focused conversations
  • Supports Python, Java, JavaScript, Go, C++, Rust, Ruby, SQL, and more
  • Trained with Axolotl, 8-bit Adam, flash attention, 8K context

What This Model Does

Llama3 8B Redmond Code290k takes LLaMA 3 8B and specializes it for code generation and explanation. Where LLaMA 3 is a general-purpose model, this variant sharpens code output quality by training on nearly 300k instruction-response pairs in the Vicuna/ShareGPT format.

Training Details

Trained with Axolotl v0.4.0:

  • Dataset: ajibawa-2023/Code-290k-ShareGPT (~290,000 conversations)
  • Learning rate: 2e-05 with cosine scheduling, 100 warmup steps
  • Epochs: 3 (2 per hyperparameters documentation)
  • Context length: 8192 tokens
  • Optimizer: 8-bit Adam with gradient checkpointing and flash attention
  • Hardware: GPU support courtesy of Redmond.ai

How to Use

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="artificialguybr/llama3-8b-redmond-code290k",
    max_new_tokens=512,
)

prompt = "Write a Python function that reverses a linked list."
result = generator(prompt)
print(result[0]["generated_text"])

The model is also endpoints compatible and works with text-generation-inference deployments.

Supported Languages

Python, Java, JavaScript, Go, C++, Rust, Ruby, SQL, MySQL, R, Julia, Haskell, and others included in the training dataset.

Limitations

  • Reduced performance on rare or niche programming languages
  • May not generalize to unseen coding styles or conventions
  • Struggles with extremely complex code or edge cases
  • Limited explanations for highly abstract or theoretical concepts

Sources