LoRA Regularization Images

Regularization images solve "language drift" — the problem where a rare token like <person_name> becomes synonymous with its entire class (like "woman"), destroying the model's ability to generate other members of that class. See https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth .

The solution is class images: normal examples of the broader category fed alongside rare-token images during training. Loss functions apply different conditioning — subject images use the rare token while class images use the plain class prompt — keeping the class embedding stable. See https://www.scottbaker.ca/AI/LoRA-Training .

LoRA is a low-rank adapter that adds small weight matrices to attention layers without modifying base model weights. Because the base model's knowledge remains intact and LoRA influence is controlled via a weight multiplier, language drift is reduced compared to full fine-tuning. See https://www.scottbaker.ca/AI/LoRA-Training .

TL;DR

• Use regularization images when you have 50+ training images or are fine-tuning entire styles/concepts (https://aituts.com/lora-training-settings/) — see LoRA training best practices
• Skip them for single-subject LoRAs to save compute and maintain precision — see LoRA basics guide
• Generate with the same model using class prompts (e.g., "a photo of a man") without the rare token — see generation settings
• Folder convention: 9_training_folder for training repeats, 1_regularization_folder for class images
• Key warning: Unrefined SDXL regularization outputs make subject images too sharp and class images unnaturally smooth (https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/) — see SDXL regularization guide

What Are LoRA Regularization Images?

Regularization images originate from the DreamBooth paper (Ruiz et al., 2022) and serve prior preservation. They prevent overfitting when fine-tuning on few images by keeping the model's understanding of broader categories intact. The DreamBooth training script can auto-generate them from the base model using class prompts (https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth ).

The prior preservation loss computes differently on subject images versus class images. On subject images, the loss is the standard denoising loss toward the subject conditioned on the rare token. On class images, the loss is the denoising loss conditioned on the plain class prompt (without the rare token). This keeps the class embedding from drifting toward the specific subject during training. See https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth and https://arxiv.org/abs/2208.12242 .

For LoRA specifically, regularization images are optional. Many experienced trainers skip them because LoRA adds small adapters rather than rewiring the model. They aren't a hard requirement and are really only necessary if you are training on a very large number of images (https://aituts.com/lora-training-settings/ ).

Worked Example: Training a "Woman" LoRA

Consider training a LoRA on a specific woman named "Jane" using the trigger token <jane>. You have 20 training images of Jane paired with captions like aa <jane> woman.

Without regularization images: the model learns <jane> → Jane's face. During inference, the prompt a photo of a woman may produce Jane's face instead of a generic woman, because the <jane> token has pulled the "woman" class embedding toward Jane's specific features. This is language drift.

With regularization images: you add 20–30 class images of other women (from Unsplash with captions like a photo of a woman) alongside the 20 Jane images. The prior preservation loss conditions the class images on the prompt a photo of a woman — without <jane>. The model learns to keep "woman" generic while still learning <jane> → Jane. See https://www.scottbaker.ca/AI/LoRA-Training .

In kohya_ss, the training images sit in 9_jane (9 repeats) and the class images in 1_woman. Training doubles in step count when regularization images are present. Example: 423 base steps become 2115 with regularization enabled. See https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ .

When Regularization Images Help vs. When They Waste Compute

Regularization images shine when you have significant data — 50+ images or are working with broader concepts like entire styles, characters, or objects. At this scale, the model needs the counterbalancing effect to prevent the subject token from absorbing the entire class distribution (https://aituts.com/lora-training-settings/ ).

For single-subject LoRAs (3-5 images), regularization images waste compute. Adding them dramatically slows training and dilutes the desired outcome. The language drift problem practically doesn't exist here, making the overhead unnecessary. See https://www.scottbaker.ca/AI/LoRA-Training .

Bad regularization images make results worse. When reg images have a different quality distribution than training images, the model associates the class prompt with that quality. Unrefined SDXL-generated reg images caused smoothed class outputs while subject images became artificially sharp. Generate reg images using the same pipeline (including refiner) as your expected output (https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ ).

How to Source or Generate Regularization Images

Three approaches, in order of preference:

1. Generate with the same base model — Use class prompts (e.g., "a photo of a man") without the rare token. This yields the most consistent results (https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth ).
2. Real photos of the same class — Unsplash or stock photos with captions. These have more diversity but may not match the model's internal distribution. Save the alt descriptions as captions for captioning consistency (https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ ).
3. Pre-made datasets — ProGamerGov's SD 1.5 regularization images on HuggingFace are a ready-made option (https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images ).

Pre-Made Dataset Options

Folder and Repeat Conventions

In kohya_ss, the Regularisation folder is optional and labeled as such in the GUI (https://aituts.com/lora-training-settings/ ). Training images sit in numbered folders like 9_training_folder (9 repeats per image), while class images use 1_regularization_folder. The blog aboutme.be illustrates this with a concrete example: folder 1_man for regularization images and 9_w0ut3r man for training images. See https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ .

A GitHub issue in kohya-ss/sd-scripts confirms that is_reg only affects loss_weights once reg images are loaded — they are handled the same as training images apart from the loss factor (https://github.com/kohya-ss/sd-scripts/issues/330 ).

CLI and TOML Configuration

The kohya_ss training script accepts the --reg_data_dir CLI flag to specify the regularization image directory. When regularization images are present, kohya_ss doubles training steps (applying a regularization factor of 2 to max_train_steps). See https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ .

For cases where reg images are silently truncated (kohya_ss caps reg images at the number of training images after repeats), the workaround is to use a TOML configuration file with class_token = instead of relying on the GUI folder field. This ensures the class token is applied consistently to regularization images. See https://github.com/kohya-ss/sd-scripts/issues/330 .

Common Errors and Fixes

Captioning Regularization Images

A point of ambiguity: captions for regularization images should use the class prompt (e.g., a photo of a woman) without the subject trigger token. If the full training caption is used (e.g., aaa 1girl <jane>), the class images leak the subject token into the prior preservation loss, defeating the purpose. When using real photos from Unsplash, save the alt descriptions as captions — these are naturally class-level descriptions without the subject keyword. See https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ .

For kohya_ss, the interaction between captions and the class_token TOML parameter is an unresolved question in the community. The kohya_ss issue #330 discusses whether reg image captions should use the folder prompt or the class prompt; the response suggested using TOML config with class_token = but no definitive answer with evidence was provided. See https://github.com/kohya-ss/sd-scripts/issues/330 .

FAQ

What Are Regularization Images in LoRA Training?

Class examples used during LoRA training to preserve the base model's understanding of broader categories. They prevent a rare token from overwriting the entire class concept in the model's knowledge. The prior preservation loss computes denoising loss on class images conditioned on the plain class prompt (without the rare token). See https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth and https://arxiv.org/abs/2208.12242 .

Do I Need Regularization Images for LoRA Training?

Most LoRA training scenarios don't need them. Only use them when training on 50+ images or broader concepts. For single-subject LoRAs, skip them to save compute. See https://aituts.com/lora-training-settings/ .

How Many Regularization Images Should I Use?

Use 200-300 according to DreamBooth research (https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth ). For LoRA specifically, the optimal number isn't well-documented — many experienced trainers skip them entirely for single-subject work. The DreamBooth guidance of num_epochs * num_samples applies to full fine-tuning; whether LoRA benefits from the same count is an open question.

Can I Skip Regularization in Kohya_ss?

Yes. The "Regularisation folder" field is optional. For single-subject LoRAs, leave it empty. This is recommended for most LoRA training scenarios (https://aituts.com/lora-training-settings/ ).

How Do I Generate Regularization Images?

Generate with the same Stable Diffusion model you're training on, using prompts that match your class but omit the rare token. Tools like kohya_ss can auto-generate these (https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth ). The CLI flag is --reg_data_dir.

What's the Difference Between Regularization and Training Images?

Training images are your specific subject (<rare_token> + subject details). Regularization images are normal class examples ("a photo of a man") that preserve the base model's understanding of that category. The only code-level difference in kohya_ss is that is_reg adjusts loss_weights for class images (https://github.com/kohya-ss/sd-scripts/issues/330 ).

Why Are My Regularization Images Making My LoRA Worse?

The model associates the class prompt with the smoothing effect from unrefined reg images, making class outputs unnaturally smooth while subject images become artificially sharp. Generate reg images using your final expected pipeline (with refiner, dpmpp_2s_ancestral) to match quality. See https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ .

Do Regularization Images Double Training Time?

Yes. When regularization images are present, kohya_ss applies a regularization factor of 2 to max_train_steps. Example from a real training run: base steps of 423 become 2115 with regularization enabled (max_train_steps (423 / 4 / 1.0 * 10 * 2) = 2115). See https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/ .

What About Style LoRAs? The behavior of regularization images for style LoRAs (artistic styles rather than specific subjects) is not documented in the available sources. The existing research covers subject LoRAs (character/person training). Whether regularization images behave differently for style LoRAs was not covered in any source reviewed.

Sources

1. Dreambooth Training: https://huggingface.co/docs/diffusers/v0.11.0/en/training/dreambooth
2. LoRA Training Guide: https://www.scottbaker.ca/AI/LoRA-Training
3. Kohya_ss LoRA Settings: https://aituts.com/lora-training-settings/
4. Regularization Impact Study: https://blog.aboutme.be/2023/08/10/findings-impact-regularization-captions-sdxl-subject-lora/
5. sd-scripts Issue #330: https://github.com/kohya-ss/sd-scripts/issues/330
6. kohya_ss Repository: https://github.com/bmaltais/kohya_ss
7. ProGamerGov Regularization Images: https://huggingface.co/datasets/ProGamerGov/StableDiffusion-v1-5-Regularization-Images
8. DreamBooth Paper (Ruiz et al., 2022): https://arxiv.org/abs/2208.12242

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "LoRA Regularization Images: When to Use Them",
  "description": "Learn when regularization images help vs. waste compute in LoRA training, how to source them, and the kohya_ss folder conventions.",
  "datePublished": "2026-07-28",
  "author": {"@type": "Person", "name": "AI Assistant"},
  "articleBody": "This guide covers when regularization images help LoRA training, when they waste compute, how to generate them, and the kohya_ss folder and repeat conventions.",
  "keywords": ["lora regularization images", "kohya_ss regularization", "dreambooth prior preservation", "class images"],
  "mainEntityOfPage": "articles/regularization-images-lora.md"
}