Train a HunyuanVideo Character LoRA

TL;DR

Use diffusion-pipe on Linux or WSL2 — native Windows is difficult/impossible for DeepSpeed pipeline parallelism (diffusion-pipe README). Prepare 30–300 images (Civitai 9967) (512×512, cropped to the character, text removed) with matching .txt captions; for video clips, diffusion-pipe extracts frames internally using single_beginning mode (diffusion-pipe README). Train at lr = 2×10⁻⁴ (diffusion-pipe README), LoRA rank 32, 500–1,500 steps (Civitai 9967) (max 30 epochs) on a single GPU; expect 1–2 hours on an RTX 40-series card (Civitai 9967). VRAM floor: ~19–20 GB at 512×512, ~23 GB at 1024×1024 with fp8 transformer, activation checkpointing, and batch size 1 (Civitai 9967). See diffusion-pipe setup.

Platform and Tooling

On Windows: enable Hyper-V in BIOS, install Ubuntu from Microsoft Store, run inside WSL2. The Runpod blog has a cloud walkthrough (Runpod blog). See also diffusion-pipe setup.

Dataset Preparation: From Video Clips to Captioned Frames

Image datasets (recommended starting point)

The community treats character LoRAs as an image training task — feed diffusion-pipe a folder of still images, each with a matching .txt caption. The trainer caches latents and text embeddings once, then trains the transformer LoRA without reloading the VAE or text encoders. See character LoRA workflow for dataset design principles.

Practical guidelines (community-reported):

• Crop to 512×512 (Civitai 9967). Upscaling to 512×512 before training is encouraged; 1024×1024 pushes VRAM higher.
• Remove all text — logos, subtitles, watermarks. Text "ruined multiple LoRAs" by baking titles into the character (Civitai 9967).
• Face-heavy coverage: prioritize clear face shots; include 1–2 images with a second person (opposite sex) so the model doesn't fuse identities (Civitai 9967).
• Aspect ratio: extreme ratios (3:1) are skipped by default. To include them, set max_ar = 2.5 in dataset.toml (Civitai 9967).

Video clip datasets

diffusion-pipe ingests video files directly via ImageIO (all formats except multi-frame WebP). Each clip is bucketed by frame count, then segmented per video_clip_mode:

Frames are VAE-encoded and cached with text embeddings. No manual frame extraction needed — point the config at .mp4/.mov files with matching .txt captions (diffusion-pipe README).

Limitation: Pre-cached embeddings mean text-encoder LoRA training is not supported (diffusion-pipe README).

Captioning: Automating the Text Files

Every media file needs a sibling .txt (e.g., frame_001.png → frame_001.txt). The community uses vision-language models for batch captioning:

Run your captioner over the dataset, write one .txt per file, verify a few samples — caption quality directly affects LoRA fidelity. See captioning guide.

Trainer Settings: diffusion-pipe Config Breakdown

The config below merges diffusion-pipe defaults with community-tuned values for reliable character LoRAs on RTX 40-series (Civitai 9967).

# main_example.toml (adapted)
output_dir = '/path/to/training_runs'
dataset = 'examples/dataset.toml'

epochs = 30
max_steps = 1500
micro_batch_size_per_gpu = 1
pipeline_stages = 1
gradient_accumulation_steps = 1
gradient_clipping = 1.0
warmup_steps = 100
lr_scheduler = 'constant'

# VRAM savers (enable all for 24 GB cards)
activation_checkpointing = 'unsloth'
blocks_to_swap = 20
PYTORCH_CUDA_ALLOC_CONF = 'expandable_segments:True'

video_clip_mode = 'single_beginning'
save_every_n_epochs = 2
checkpoint_every_n_minutes = 120

[model]
type = 'hunyuan-video'
transformer_path = '/path/to/hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors'
vae_path = '/path/to/hunyuan_video_vae_bf16.safetensors'
llm_path = '/path/to/llava-llama-3-8b-text-encoder-tokenizer'
clip_path = '/path/to/clip-vit-large-patch14'
dtype = 'bfloat16'
transformer_dtype = 'float8'
timestep_sample_method = 'logit_normal'

[adapter]
type = 'lora'
rank = 32
dtype = 'bfloat16'

[optimizer]
type = 'adamw_optimi'
lr = 2e-4
betas = [0.9, 0.99]
weight_decay = 0.01
eps = 1e-8

Launch (RTX 40-series) (Civitai 9967):

NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 \
deepspeed --num_gpus=1 train.py --deepspeed --config your_config.toml

VRAM Requirements: Training vs. Inference

Training fits on 24 GB while inference needs 30–60 GB because training runs at 512×512 (not 720p), processes one sample at a time, and keeps only the transformer LoRA weights in GPU. FP8 quantization, unsloth checkpointing, and block swapping reduce memory further. Inference must hold the full transformer, VAE, and both text encoders at target resolution (Civitai 9967; HunyuanVideo GitHub).

Common Errors and Fixes

FAQ

How much VRAM do I need to train a HunyuanVideo LoRA?

19–20 GB at 512×512 with FP8 transformer, unsloth checkpointing, and block swapping. A 24 GB card (RTX 3090/4090) has comfortable headroom; 1024×1024 pushes to ~23 GB with little margin (Civitai 9967).

Can I train on a 24 GB GPU?

Yes. Community reports success on RTX 3090/4090 (24 GB) at 512×512. Enable all three VRAM savers: transformer_dtype = 'float8', activation_checkpointing = 'unsloth', blocks_to_swap = 20 (diffusion-pipe README).

How do I extract frames from video for LoRA training?

You don't — diffusion-pipe does it internally. Place .mp4/.mov files in your dataset folder with matching .txt captions, set video_clip_mode = 'single_beginning' (default), and the trainer extracts, encodes, and caches frames automatically (diffusion-pipe README).

What captioning model should I use?

Florence-2 is fastest and most common. Llama-Vision gives similar quality. For video clips, Meta Apollo works well on short segments (Civitai 9967). All three are VLMs you run locally or via API to produce one .txt per media file.

What is the minimum dataset size?

30 images is the community floor (Civitai 9967). Up to 300+ works but each image increases caching time and steps needed.

How long does training take?

1–2 hours on RTX 40-series for 500–1,500 steps at 512×512. Much of that is the one-time latent/text caching pass; re-runs with the same dataset skip caching via --trust_cache (diffusion-pipe README).

What learning rate should I use?

Default is 2×10⁻⁵; community uses 2×10⁻⁴ (10× higher) for character LoRAs with faster convergence. Start at 2×10⁻⁴ with warmup_steps = 100 (diffusion-pipe README).

Does diffusion-pipe work on Windows?

Not natively. DeepSpeed pipeline parallelism is difficult/impossible on native Windows. Use WSL2 (Ubuntu) with Hyper-V enabled, or Linux/cloud. The Runpod blog has a cloud guide (Runpod blog). For more help, see VRAM optimization tips.

For further reading on model optimization, see LoRA training best practices.

Sources