Qwen2.5 0.5B OpenHermes 2.5: Lightweight Instruct Model
Qwen2.5 0.5B fine-tuned on OpenHermes 2.5 for instruction following. Apache 2.0, runs locally, 0.5B params.
Qwen2.5 0.5B OpenHermes 2.5
A lightweight 0.5B parameter language model fine-tuned on the OpenHermes 2.5 dataset for instruction following and chat tasks.
What This Model Does
This model takes Qwen/Qwen2.5-0.5B — a compact base model with strong coding, math, and multilingual capabilities — and fine-tunes it on 1 million synthetic instruction and chat samples from OpenHermes 2.5. The result is a model that follows instructions better than the base while staying small enough to run on consumer hardware.
Model Specifications
Property | Value
|---|---
Base model | Qwen/Qwen2.5-0.5B Parameters | 0.49B (0.36B non-embedding) Architecture | Transformers, RoPE, SwiGLU, RMSNorm, GQA (14Q/2KV) Context length | 32,768 tokens License | Apache 2.0 Framework | Transformers >= 4.37.0
Training Details
- Dataset: teknium/OpenHermes-2.5 (1M samples)
- Epochs: 3
- Learning rate: 1e-5 (cosine, 100 warmup steps)
- Batch size: 40 (grad accum 8)
- Precision: BF16
- Sequence length: 4096 with sample packing
- Optimizer: Adam (β=(0.9,0.999), ε=1e-8)
- Weight decay: 0.01
- Built with: Axolotl 0.4.1
How to Use
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"artificialguybr/Qwen2.5-0.5B-OpenHermes2.5",
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
"artificialguybr/Qwen2.5-0.5B-OpenHermes2.5"
)For chat, apply the Qwen2.5 chat template via the tokenizer.
Limitations
- Not a chat model out of the box — needs SFT/RLHF for production conversational use
- Performance varies by language and domain
- May reflect biases from OpenHermes 2.5 training data