JoyCaption, TagGUI, and AICaptionHelper

TL;DR

• JoyCaption Alpha 2 processes 21 images in 45 seconds on a 3090
• TagGUI runs on GPU or CPU with a graphical interface
• AICaptionHelper offers interactive OpenAI API integration

Why This Matters

Training Stable Diffusion models requires datasets of images paired with accurate text descriptions. During training, the text description gets encoded into the model attached to the vector data, so that Stable Diffusion can use that text (or variations) to generate images later. Manually writing captions for hundreds or thousands of images is tedious, and the quality of those captions directly affects model behavior.

Recent open-source VLMs have changed the equation. Models like CogVLM and LLaVA now produce captions with enough descriptive depth to leverage SDXL's text encoding capabilities — capturing emotions, lighting, composition, and mood rather than just naming objects. Two batch processing approaches have emerged: CLI scripts that run unattended over a directory of images, and desktop applications that let you review and edit results interactively.

Three Main Solutions

The three tools differ in cost, workflow, and hardware requirements. JoyCaption and TagGUI are free and open-source, while AICaptionHelper runs in a browser with optional OpenAI API costs. For related workflows, see automated dataset building and benchmark ComfyUI setup.

JoyCaption Alpha 2

Architecture and Setup

JoyCaption Alpha 2 runs on the fancyfeast/llama-joycaption-alpha-two-hf-llava model hosted on Hugging Face. The model combines meta-llama/Llama-3.1-8B-Instruct as the language backbone with google/siglip-so400m-patch14-384 as the vision encoder. The batch script uses LlavaForConditionalGeneration from Hugging Face Transformers.

Setup requires cloning the joy-caption-batch repository and running the batch script (batch-alpha2.py). The script processes images from an /input subdirectory by default, configurable with --input-dir.

For related workflows, see character-consistency workflow.

Performance

On a 3090 with 24GB VRAM, batch count 8 processes 21 images in about 45 seconds, versus 3 minutes at batch size 1 — roughly a 4× improvement (source). Default sampling parameters are temperature 0.5 and top-k 10 with top-p 0.9.

Standard mode needs 19–22GB VRAM. The Pre-Alpha version offers a low VRAM mode that loads a quantized llama3-8b-bnb-4bit variant from Unsloth, bringing the requirement down to about 10GB. Low VRAM mode is not available in Alpha 2. With Pre-Alpha in low VRAM mode on a 3090, batch count can go as high as 16.

Prompt Selection

The script ships with several built-in prompt templates. The default — "Please provide a detailed description of the image" — produces long-form prose captions. Other options include short descriptions, Stable Diffusion prompts, MidJourney prompts, Booru-style tag lists, and social media captions. You can also load a custom JSON prompt file with weighted entries.

File I/O

The script writes .txt caption files alongside each image by default. Output format can be set to txt or json. Additional flags include --recursive (process subfolders), --prepend-string and --append-string (wrap every caption), and --overwrite (replace existing files). See fpgaminer/joycaption for the original batch script.

TagGUI

Model Support

TagGUI is a cross-platform desktop application for tagging and captioning image datasets. It connects to Hugging Face model hubs and supports multiple captioning backends including BLIP-2, LLaVA, and CogVLM — the latter detailed on the THUDM/cogvlm2-llama3-chat-19B model card. The Civitai user review highlights CogVLM's detailed caption quality for 90-image datasets.

CogVLM2 handles up to 8K tokens and images up to 1344×1344 pixels, scoring 84.2 on TextVQA. Users can point TagGUI at a local model directory in Settings to avoid repeated downloads. First-time model loads take several minutes; subsequent loads are much faster.

Batch Captioning

Select images in the image list, pick a model in the Auto-Captioner pane, and click Start Auto-Captioning. The application downloads models on first use and caches them locally. The prompt field accepts template variables: {tags} inserts existing tags, {name} inserts the filename, and {directory} inserts the folder name. You can also set words to discourage or require in the output.

Tag Management

TagGUI reads .txt files matching image filenames and auto-saves every edit. The filtering box supports prefixes: tag: for tag matching, caption: for caption text search, name: for filename patterns, and path: for full directory paths. Numeric filters work too — tags:>10 finds images with more than 10 tags; tokens:<75 finds captions under 75 tokens.

Combined with AND, OR, and NOT operators plus parentheses for precedence, this lets users build queries like tag:cat AND tokens:<=75 to find short-tagged cat images. Keyboard shortcuts speed up bulk work: Ctrl+J jumps to the first untagged image, Ctrl+R opens Find and Replace, and Ctrl+B opens Batch Reorder Tags. For character consistency during training, see character-consistency workflow.

AICaptionHelper

AICaptionHelper is hosted on Civitai. It combines user-provided keywords with optional OpenAI API refinement to produce captions. Users describe their images across structured fields — subject, hairstyle, activity, location, expressions — and the tool assembles a caption from those inputs. An OpenAI API key is optional; without one, the tool saves exactly what you type. With one, the OpenAI API refines spelling, grammar, and natural language flow.

No independent GitHub repository or standalone documentation was found beyond the Civitai article, so details about its internals are limited to that source.

Common Errors and Fixes

Frequently Asked Questions

How much VRAM does JoyCaption need? Alpha 2 needs 19–22GB. Pre-Alpha with low VRAM mode needs about 10GB (source).

Can I batch process images with JoyCaption? Yes. Drop images in /input and run the batch script. Configure batch count with --batch-processing-count. On a 3090, batch 8 gives roughly 4× the throughput of batch 1.

What models does TagGUI support? BLIP-2, LLaVA, CogVLM, and others available through Hugging Face. GPU and CPU generation are both supported (source).

Does TagGUI work offline? Yes, after the first model download. Models are cached locally.

What image formats work with these tools? JoyCaption supports JPG, JPEG, PNG, BMP, GIF, TIFF, and WebP. TagGUI has similar format support through PIL/Pillow.

Sources

• MNeMoNiCuZ/joy-caption-batch — batch scripts, setup instructions, VRAM notes
• jhc13/taggui — TagGUI desktop application
• fpgaminer/joycaption — original JoyCaption batch script
• fancyfeast/llama-joycaption-alpha-two-hf-llava — model card
• THUDM/cogvlm2-llama3-chat-19B — CogVLM2 benchmarks
• Civitai: AI Caption Helper — AICaptionHelper documentation
• Civitai: Effortless Image Captioning — TagGUI user review
• Civitai: JoyCaption Alpha 2 Update — JoyCaption batch update notes