ArtificialGuyBR

Home / Blog / Training LoRAs

Captioning Tools 2026: JoyCaption vs WD Tagger vs Manual

Compare JoyCaption Beta One's 12 caption styles and word-count control against WD Tagger's ONNX batch inference, plus manual tagging workflows that scale.

10 sources cited Training LoRAs

On this page

  1. TL;DR
  2. Why captioning tool choice matters
  3. What is JoyCaption Beta One?
  4. JoyCaption caption styles and length control
  5. WD Tagger v3: architectures and batch inference
  6. Batch tooling comparison
  7. Common errors and fixes
  8. FAQ
  9. Sources
  10. Internal links

TL;DR

Why captioning tool choice matters

Training diffusion models lives or dies by caption quality. A model trained on vague, repetitive, or censored captions will hallucinate, ignore prompts, or refuse NSFW concepts. The three mainstream paths — VLM captioning (JoyCaption), tagger ensembles (WD Tagger), and human-in-the-loop — each hit different points on the speed/quality/cost triangle.

What is JoyCaption Beta One?

JoyCaption Beta One is a Visual Language Model (VLM) built on LLaVA architecture: a SigLIP2-SO400M vision encoder fused to Llama-3.1-8B-Instruct. The model weights are Apache-2.0 licensed, free to download, and run locally. Unlike commercial VLMs, it was trained with explicit SFW/NSFW balance and no "cylindrical object" euphemisms.

Key specs: Parameter | Value | Source

|-----------|-------|--------

Base LLM | meta-llama/Llama-3.1-8B-Instruct | https://huggingface.co/fancyfeast/llama-joycaption-beta-one-hf-llava Vision encoder | google/siglip2-so400m-patch14-384 | https://huggingface.co/fancyfeast/llama-joycaption-beta-one-hf-llava Native dtype | bfloat16 | https://github.com/fpgaminer/joycaption VRAM (bfloat16) | ~17 GB | https://github.com/fpgaminer/joycaption Quantization | 8-bit, 4-bit NF4 (double quant) | https://github.com/fpgaminer/joycaption_comfyui/blob/main/nodes.py Training samples (Beta One) | 2.4 million | https://civitai.com/articles/14672 DPO rounds | 2 (10k + 20k preference pairs) | https://civitai.com/articles/14672 Performance metrics: Metric | Value | Source

|--------|-------|--------

Glitch rate (overall) | 1.5% after DPO | https://civitai.com/articles/14672 Glitch rate (SD Prompt) | ~3% | https://civitai.com/articles/14672 Human evaluation accuracy | 67% (Beta One) | https://civitai.com/articles/14672 Human evaluation accuracy | 55% (Alpha Two, GPT-4o) | https://civitai.com/articles/14672 Tag augmentation score | 6.51/10 (SOTA VLM judge) | https://civitai.com/articles/14672

JoyCaption caption styles and length control

The ComfyUI node exposes 12 caption types and a dual-mode length selector.

Caption type menu

Caption type | Use case

|--------------|----------

Descriptive | Dataset captions, alt text Descriptive (Casual) | Human-readable descriptions Straightforward | Middle ground Stable Diffusion Prompt | Direct prompting (mimics user prompts) MidJourney | MJ-style artistic modifiers Danbooru tag list | Booru-style training (category:tag) e621 tag list | e621-format training Rule34 tag list | Rule34-format training Booru-like tag list | Generic booru training Art Critic | Analysis/feedback Product Listing | E-commerce captions Social Media Post | Instagram/Twitter alt text

Length control

The selector accepts descriptive labels (very short … very long) or exact word counts (20, 30, … 260 in steps of 10). This gives you deterministic template filling for both casual and technical use cases.

WD Tagger v3: architectures and batch inference

The "Datasets for Dummies" guide (Civitai 2217) describes a hybrid pipeline: scrape boorus, AI-tag everything with WD Tagger (via Kohya-ss wd14_tagger.py or Holostrawberry's Dataset Helper Colab), merge tag sets, then run a full manual QA pass per file. Teams parallelize across annotators with a shared style guide. Time estimate: ~30–60 s/image. The "Datasets for Dummies" guide describes a hybrid pipeline:

  1. Scrape from boorus — pre-tagged, but noisy.
  2. AI-tag everything — run WD Tagger (via Kohya-ss wd14_tagger script or Holostrawberry's Dataset Helper Colab) even on booru images.
  3. Merge tag sets — PowerShell/Notepad++ concatenation, deduplicate, alphabetize.
  4. Blacklist count/relation tags — 1girl, 2girls, multiple girls, brother and sister, etc. (tagger can't count relationships).
  5. Manual QA pass — every file, confirm/remove/add tags. "Tag everything: every clothing item, eye color, hair color."

Time estimate: ~30–60 s/image for thorough review, but produces the cleanest training sets. Teams parallelize the manual pass across annotators.

Batch tooling comparison

Tool | Batch mode | Throughput | Hardware | Best for

|------|------------|------------|----------|----------

JoyCaption (vLLM) | OpenAI-compatible API, prefix caching | ~2–4 img/s | 24 GB+ GPU | Mixed-content, natural language JoyCaption (ComfyUI) | Node loop, quantization | ~1–2 img/s (4-bit) | 12–16 GB GPU | Local experimentation, style variety WD Tagger ONNX | ONNX model with unfixed batch dim | 50–200 img/s | 8–16 GB GPU / CPU | Pure tag extraction, anime/booru Kohya-ss wd14_tagger.py | Directory walk, sequential | ~5–10 img/s | 8 GB+ GPU | Drop-in dataset prep, integrates with training Manual (team) | Human parallel | ~1–2 img/min/person | Any | Quality-critical, small/medium datasets

Note: JoyCaption vLLM (vllm serve fancyfeast/llama-joycaption-beta-one-hf-llava --max-model-len 4096 --enable-prefix-caching) is the only production-grade path for natural language captioning.

Common errors and fixes

Symptom | Cause | Fix

|---------|-------|-----

JoyCaption OOM on 24 GB | bfloat16 model + KV cache > VRAM | Use 4-bit NF4 quantization or vLLM with --gpu-memory-utilization 0.85 JoyCaption repeats tags in Booru mode | Pre-DPO model or high temperature | Upgrade to Beta One; set temperature=0.3, top_p=0.9 WD Tagger misses character tags | Character tags filtered at <600 images | Add known characters manually; use JoyCaption tag augmentation processor.apply_chat_template produces duplicate &lt;bos&gt; | Wrong chat template combo | Use snippet from HF model card: exact apply_chat_template and processor calls Kohya-ss tagger writes duplicate newlines | PowerShell merge script artifact | Run sed -i &#39;s/\r\n/, /g&#39; *.txt after merge, then deduplicate

FAQ

What caption style for SDXL fine-tuning? Straightforward or Stable Diffusion Prompt mode. Straightforward gives clean natural language (~50–100 words) that SDXL understands well. SD Prompt mode mimics user prompts but has a 3% glitch rate and lower variety after DPO.

Can JoyCaption run on 12 GB VRAM? Yes, with 4-bit NF4 quantization (ComfyUI "Maximum Savings" preset). Expect ~1–2 img/s. For batch workloads, use vLLM on a 24 GB+ card.

Which WD Tagger backbone is better: ViT or SwinV2? SwinV2 (wd-swinv2-tagger-v3) scores higher F1 (0.4541 vs 0.4402) and has 7× more downloads. For anime/booru, prefer SwinV2; for photorealistic, ViT may generalize better.

How do I merge booru tags with AI tags without duplicates? The Civitai guide uses PowerShell to concatenate same-named .txt files from two folders, then Notepad++ regex replace \r\n → , , then deduplicate. Kohya-ss's wd14_tagger has --undesired_tags to filter specific tags; deduplication is done post-merge with text tools.

Does JoyCaption support tag augmentation? Yes, Beta One adds tag augmentation: pass comma-separated booru tags and the model integrates them (judged 6.51/10 by a SOTA VLM). Use extra_options: &quot;tag_augmentation: tag1, tag2, tag3&quot; in the ComfyUI node.

What's the glitch rate for JoyCaption Beta One? Overall 1.5% after DPO (SD Prompt: ~3%). Commercial VLMs (GPT-4o) report ~0.01%.

Can I use WD Tagger ONNX on CPU? Yes. onnxruntime CPU provider works; throughput ~5–10 img/s, no GPU required. Install onnxruntime≥1.17.0 and load the .onnx file.

How do I evaluate caption quality automatically? No standard automated metric exists. The JoyCaption author uses a brutal hand-eval: +1 per accurate detail, −1 per inaccurate detail, normalized against a human-written "perfect" caption. Beta One scores 67% vs Alpha Two 55% vs GPT-4o 55%.

Sources

  1. JoyCaption GitHub — architecture, VRAM, licensing
  2. JoyCaption ComfyUI nodes — caption types, length choices
  3. JoyCaption HF model card — specs, usage snippet
  4. WD ViT Tagger v3 model card — training data, validation, ONNX
  5. WD SwinV2 Tagger v3 model card — architecture, metrics
  6. WD v1-4 model card — legacy reference
  7. Kohya-ss sd-scripts README — wd14_tagger link
  8. Civitai: JoyCaption Beta One — training samples, DPO, benchmarks
  9. Civitai: Datasets for Dummies — hybrid tagging workflow
  10. Civitai: Prompts Creator — paid manual tagging tool