How LoRAs Work and How to Use Them in Stable Diffusion

A LoRA (Low-Rank Adaptation) is a small patch that modifies Stable Diffusion models without replacing them. Instead of downloading an entirely new checkpoint, you load a LoRA file that injects trainable adjustments into specific layers. This article explains the math behind LoRAs, how to control their strength, and how to match them to your base model.

For hands-on guidance on preparing your first dataset and training, see our train your first LoRA free tutorial. If you're building a style library, our LoRA library management guide covers organization and versioning. For SDXL-specific tips, see SDXL photoreal LoRA tips.

TL;DR

• Math: The key formula is $h = W_0 x + BAx$, where $W_0$ is frozen original weight and $BA$ is trainable update
• Strength control: Inference-time weight (e.g., <lora:model:0.7>) controls LoRA effect
• Stacking: Multiple LoRAs sum their effects (chain ComfyUI nodes or use set_adapters in Diffusers)
• Compatibility: Match LoRA architecture to base model dimensions (SDXL vs SD1.5)
• Installation: models/loras/ for ComfyUI, models/Lora/ for AUTOMATIC1111

How LoRA Works at Inference Time

The Mathematical Foundation

When you load a LoRA, you are not replacing base model weights. You are adding small, trainable adjustments to them.

The update follows a simple formula:

$$h = W_0 x + BAx$$

Each term:

• $W_0$: The original weight matrix from your base model. This stays frozen—never changes during training or inference.
• $x$: Input activations (from text embeddings or intermediate layers).
• $B$: First low-rank matrix. Shape: output dimension × rank (e.g., dmodel × r).
• $A$: Second low-rank matrix. Shape: rank × input dimension (e.g., r × dmodel).
• $BA$: The trainable update. Multiplying these gives a rank-r approximation of the full-weight update that would be trained in fine-tuning.

The original LoRA paper found that model weights live on a low-intrinsic-dimension manifold, meaning weight updates can be represented with far fewer parameters than the original weights.

"We hypothesize that the change in weights during model adaptation also has a low 'intrinsic rank' during adaptation." — https://arxiv.org/abs/2106.09685

Why This Matters for Stable Diffusion

For Stable Diffusion's UNet attention layers, standard weight matrices are full-rank (e.g., 768×768 for SDXL). Instead of updating 768² parameters, LoRA updates only r×768 + 768×r = 1536r parameters.

With r = 8, that is 12,288 trainable parameters versus 589,824 full-rank weights—a 48× reduction.

"At inference: zero additional latency versus a fully fine-tuned model by construction." — https://arxiv.org/abs/2106.09685

Most UIs permanently merge the LoRA into the base model during loading. The $W_0 + BA$ update is computed once and stored as new weights. This eliminates runtime overhead—merged weights are normal model parameters with no additional memory cost.

Strength Controls: Alpha versus Weight

Confusing "alpha" and "weight" is the most common mistake when using LoRAs. They operate at different levels:

Training-time alpha (α):

• Part of training hyperparameters (kohya-ss: --network_alpha, default 1; PEFT: lora_alpha, default equals rank)
• Controls learning rate scaling during training: (α/r) scales the update
• Reference: https://github.com/kohya-ss/sd-scripts/blob/main/docs/train_network.md

Inference-time weight (the number in <lora:model:1>):

• Runtime multiplier applied to the entire ΔW = BA update
• Weight = 1 → full LoRA effect
• Weight = 0 → no effect (pure base model)
• Weight = 0.5 → half the effect

The original paper establishes scaling as (α/r)·BA. Different implementations handle this differently—the cloneofsimo repository suggests the inference weight replaces alpha scaling entirely, while other implementations multiply both.

How Multiple LoRAs Stack

When you apply multiple LoRAs, their effects sum:

$$h = W_0 x + \frac{\alpha_1}{r_1} B_1 A_1 x + \frac{\alpha_2}{r_2} B_2 A_2 x$$

If two LoRAs both target the same attention layer (e.g., both modify Wq), their updates add together.

Chaining LoRAs

ComfyUI: Chain LoraLoader nodes sequentially:

LoraLoader node 1 → LoraLoader node 2 → [...]

Each node adds its own low-rank matrices to the base weights. Reference: https://comfyanonymous.github.io/ComfyUI_examples/lora/

Diffusers: Use set_adapters() for weighted control:

pipeline.set_adapters(["style1", "style2"], adapter_weights=[0.7, 0.8])

Advanced Merging: PEFT's add_weighted_adapter() supports TIES and DARE methods for offline combination.

"LoRA is orthogonal to many other parameter-efficient methods and can be combined with many of them, such as prefix-tuning." — https://arxiv.org/abs/2106.09685

Matching LoRAs to Base Models

LoRA matrices are shaped to match the architecture dimensions of the base model they were trained on. An SD1.5 LoRA has B/A matrices sized for SD1.5's UNet attention dimensions; an SDXL LoRA has larger dimensions.

Architecture mismatch examples:

• SDXL LoRA → SD1.5 base: ❌ matrix dimension mismatch
• SD1.5 LoRA → SD1.5 base: ✅ dimensions match

"LoRAs also need to be used with another model." — https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters

Different base models also have varying numbers of attention layers and may target different layer types (self-attention versus cross-attention).

Where to Install LoRA Files

Reference: https://comfyanonymous.github.io/ComfyUI_examples/lora/

Common Errors and Fixes

Frequently Asked Questions

How much VRAM does a LoRA add? The LoRA matrices themselves are small (1–6 MB for low-rank, up to ~200 MB for higher ranks). The base model loads normally. After merging, there is no additional VRAM cost.

Can I apply LoRAs to non-attention layers? In principle, LoRA can target any weight matrix. In practice, most SD LoRAs target attention layers only (Wq, Wk, Wv, Wo). kohya-ss supports Conv2d layers via LoRA-C3Lier (reference: https://github.com/kohya-ss/sd-scripts/blob/main/docs/train_network.md).

What is the optimal rank for LoRAs? The original paper tested ranks 4–8 for language models. For SD, ranks 8–128 are common; lower ranks (4–16) trade quality for speed and VRAM efficiency.

How many LoRAs can I stack? No hard limit. Each LoRA adds small matrices to the same layers. Diminishing returns appear as more LoRAs modify the same weight matrices, but practical stacking of several LoRAs is common.

Do I need special software to use LoRAs? Any UI that supports LoRAs works—AUTOMATIC1111, ComfyUI, Forge, and Diffusers all handle them natively.

How do I know if a LoRA will work with my base model? Check the model card or description. Most list compatible base models (e.g., "SDXL v1.0 base", "SD 1.5"). When in doubt, inspect the .safetensors file dimensions.

Sources