SDXL LoRA Training Guide: Lessons from 10 Released Models

This SDXL LoRA training guide synthesizes lessons from ten creators who published photorealistic LoRAs on Civitai. SDXL LoRAs offer higher quality than SD1.5 but are less forgiving — dataset choice, caption style, and hyperparameters matter more.

TL;DR

• Use ViT-g-14-laion2B-s34B-b88K for SDXL captioning via clip-interrogator-ext
• Minimum 30 diverse images (50 preferred) at ≥1024×1024 resolution
• Network rank: 256/128/64 for likeness, 16/32 for style/concept
• Learning rate: 0.0012 for poses/concepts, 2e-6 for likeness (same for TE/UNET)
• Constant Scheduler + AdamW8Bit; stop at epoch 5 for likeness
• Add FFHQ regularization to preserve general face knowledge
• Sample prompts = image captions; max 768×1024 resolution

SDXL LoRA Training Guide: Core Principles

SDXL was trained on 1024×1024 images with two text encoders: CLIP ViT-L/14 and OpenCLIP ViT-bigG/14. This gives better prompt alignment than SD1.5 but makes LoRAs more sensitive to data quality, caption precision, and hyperparameters.

SD1.5's smaller capacity tolerated sloppier data but produced less detail. SDXL LoRAs demand more attention but deliver superior results when trained correctly. For architectural background, see our SDXL architecture explained guide.

Choosing Your Training Dataset

Resolution Requirements

SDXL's native resolution is 1024×1024. Training images should match or exceed this.

• Scale the smallest dimension to exactly 1024 pixels
• Preserve aspect ratio — portrait ~1024×1536px, landscape ~1536×1024px
• Windows users: Microsoft Powertoys adds right-click resizing presets

Dataset Size and Diversity

Small datasets (<30 images) produce "leaky weights" where undesired attributes combine. Two independent creators report 30 images as their minimum for photorealistic likeness models.

One creator's documented results with 30 images:

• 20 repeats/epoch (600 samples/epoch)
• 22 minutes total on RTX 4090
• Epoch 5 optimal; overtraining after

Consensus:

• Minimum: 30 diverse images
• Recommended: 50+ for robustness
• See our training dataset preparation checklist

Regularization Images for Likeness Training

Regularization images prevent "language drift" — losing general class knowledge. Technique from DreamBooth paper (Ruiz et al., 2022) arXiv:2208.12242.

FFHQ provides 70,000 diverse 1024×1024 faces — ideal for photorealistic likeness. Dataset includes diversity in age, ethnicity, and background.

git clone https://github.com/NVlabs/ffhq-dataset
cd ffhq-dataset
python download_ffhq.py -i

Caption FFHQ images using the same method as your primary dataset.

Generating Accurate Captions

Why Tag-Based Captioning Matters for SDXL

Tag-based captioning aligns with SDXL's internal representation because LAION-B uses comma-separated tags alongside natural language. BLIP generates free-form descriptions; WD14 taggers produce the comma-separated format.

During LoRA training, tag-based captions give explicit concept control and reduce hallucinations. Most SDXL LoRA creators prefer tag-based approaches for consistency.

Recommended Captioning Tools

ViT-g-14-laion2B-s34B-b88K via CLIP Interrogator:

One source identifies this CLIP model as aligned with SDXL's pretraining text encoder. The clip-interrogator-ext extension for Stable Diffusion WebUI provides batch captioning. On ≤24GB VRAM, unload SD checkpoint to RAM first. Use "fast" mode — manual review rarely needed.

WD14 SmilingWolf Models:

Two confirmed models on Hugging Face support ratings, characters, general tags:

Tagger v3 trained on Danbooru images. One creator runs all three SmilingWolf models simultaneously and uses Kohya-ss's "Append TAGs" to avoid duplicates.

Sample Prompts During Training

Using identical prompts to captions saves time and diagnoses data issues. One source recommends:

• Copy captions from 2–4 training images as samples
• Same seed, CFG, resolution for each
• No negative prompt
• Max 768×1024 — higher causes swap/slowdown

Training Hyperparameters

Learning Rates

One source reports wide range by LoRA type:

Pose/Concept (0.0012): Higher LR for abstract concepts/styles; train <2,000 steps Likeness (2e-6): Lower LR for subjects; train >1,500 steps

Text Encoder & UNET: Set both LRs equal to chosen LR for balanced adaptation.

Optimizer and Scheduler

One source finds Constant Scheduler + AdamW8Bit most consistent. Constant maintains fixed LR; AdamW8Bit is memory-efficient, faster on consumer hardware.

Network Configuration

Network rank determines capacity. One source's recommendations:

Alpha=1 across all configs.

Settings Quick Reference

SDXL to SD1.5 Porting

Downscale training data by 50%, apply SD1.5 settings. Lower quality than native SDXL but easier — useful for older hardware/faster generation. See our SDXL to SD1.5 migration guide.

Common Errors and Fixes

VRAM Management

Problem: Captioners slow/crash Fix: WebUI Settings → Actions → "Unload SD Checkpoint to RAM" before captioning (≤24GB VRAM)

Sample Prompt Mismatch

Problem: Samples don't match training images Fix: Manually verify 2–4 captions; adjust dataset

Overfitting/Overtraining

Problem: Only exact training examples generated Fix: More FFHQ regularization, fewer repeats, stop at epoch 3–4

Resolution Swapping

Problem: Training stalls at high sample resolution Fix: Cap samples at 768×1024; monitor VRAM

FAQ

What learning rate for SDXL LoRA training?

Pose/concept: 0.0012, <2,000 steps. Likeness: 2e-6, >1,500 steps. TE/UNET LR = chosen LR.

How many images for photorealistic SDXL LoRA?

Minimum 30 diverse. Best: 50+. Portraits need more diversity; style/concept work with fewer.

BLIP or WD14 tagger for SDXL captioning?

WD14 taggers produce comma-separated tags matching SDXL's training format. SmilingWolf models widely used. CLIP Interrogator with ViT-g-14-laion2B-s34B-b88K is alternative.

Best optimizer for SDXL LoRA training?

Constant Scheduler + AdamW8Bit — most commonly recommended among sources.

How to prevent overtraining?

Monitor sample quality. Likeness: stop at epoch 5 per one source. Degradation → restart with fewer repeats/lower LR. FFHQ regularization helps.

What resolution for SDXL training images?

Minimum 1024×1024. Smallest dimension = 1024px, preserve aspect ratio (portrait ~1024×1536px, landscape ~1536×1024px).

Regularization needed for style/concept LoRAs?

One source: helpful for likeness, "much less necessary for other categories."

Sources