LLM · Text Generation
Qwen2.5 0.5B OpenHermes 2.5
Qwen2.5 0.5B fine-tuned on OpenHermes 2.5 for instruction following, coding, and chat; Apache 2.0 licensed for practical local inference.
About this model
This model is a fine-tuned version of Qwen/Qwen2.5-0.5B trained on the OpenHermes 2.5 dataset (1M synthetic instruction/chat samples). It inherits Qwen2.5's improvements in coding, mathematics, instruction following, long-context support (32K tokens), and multilingual capabilities across 29+ languages. The model uses Transformers architecture with RoPE, SwiGLU, RMSNorm, and grouped-query attention (14 Q heads, 2 KV heads).
Training: 3 epochs, learning rate 1e-5, cosine scheduler with 100 warmup steps, batch size 40 (gradient accumulation 8), BF16 mixed precision, sequence length 4096, sample packing enabled. Built with Axolotl framework on Transformers 4.45.0, PyTorch 2.3.1.
License: Apache 2.0. Requires transformers >= 4.37.0. Not recommended for direct conversational use without further SFT/RLHF post-training.
Learn more
Project signals
- 202 Hugging Face downloads
- 4 Hugging Face likes
Topics
transformers · pytorch · safetensors · qwen2 · text-generation · conversational · dataset:teknium/openhermes-2.5 · base_model:qwen/qwen2.5-0.5b · base_model:finetune:qwen/qwen2.5-0.5b · text-generation-inference