ArtificialGuyBR

Home / Blog / Fixing things

Qwen2.5 0.5B OpenHermes 2.5: Lightweight Instruct Model

Qwen2.5 0.5B fine-tuned on OpenHermes 2.5 for instruction following. Apache 2.0, runs locally, 0.5B params.

By ArtificialGuyBR Published Updated 1 sources cited Fixing things

A lightweight 0.5B parameter language model fine-tuned on the OpenHermes 2.5 dataset for instruction following and chat tasks.

What This Model Does

This model takes Qwen/Qwen2.5-0.5B — a compact base model with strong coding, math, and multilingual capabilities — and fine-tunes it on 1 million synthetic instruction and chat samples from OpenHermes 2.5. The result is a model that follows instructions better than the base while staying small enough to run on consumer hardware.

Model Specifications

Property | Value

|---|---

Base model | Qwen/Qwen2.5-0.5B Parameters | 0.49B (0.36B non-embedding) Architecture | Transformers, RoPE, SwiGLU, RMSNorm, GQA (14Q/2KV) Context length | 32,768 tokens License | Apache 2.0 Framework | Transformers >= 4.37.0

Training Details

  • Dataset: teknium/OpenHermes-2.5 (1M samples)
  • Epochs: 3
  • Learning rate: 1e-5 (cosine, 100 warmup steps)
  • Batch size: 40 (grad accum 8)
  • Precision: BF16
  • Sequence length: 4096 with sample packing
  • Optimizer: Adam (β=(0.9,0.999), ε=1e-8)
  • Weight decay: 0.01
  • Built with: Axolotl 0.4.1

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "artificialguybr/Qwen2.5-0.5B-OpenHermes2.5",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "artificialguybr/Qwen2.5-0.5B-OpenHermes2.5"
)

For chat, apply the Qwen2.5 chat template via the tokenizer.

Limitations

  • Not a chat model out of the box — needs SFT/RLHF for production conversational use
  • Performance varies by language and domain
  • May reflect biases from OpenHermes 2.5 training data

Sources