Merging LoRAs vs Stacking

TL;DR

• Merging combines multiple LoRA weight matrices into a single adapter before inference — one forward pass, smaller VRAM footprint, faster generation.
• Stacking loads each LoRA separately and sums their contributions during every forward pass — preserves individual optimizations but uses more memory and compute.
• Merge when: you need deployment simplicity, strict VRAM limits, or a single distributable file. Stack when: maximum quality per adapter matters and you can afford the overhead.
• Quality cost: Merging can lose fine-grained adaptations because each LoRA was trained independently; no global optimum is guaranteed.

What "Merging" and "Stacking" Actually Mean

A LoRA adapter is a pair of low-rank matrices (lora_down and lora_up, or A and B) representing a change to a base model's weight matrix: ΔW = B × A. The rank r is much smaller than the original dimensions, keeping the adapter tiny — a small fraction of the base model's size.

Most UIs (Automatic1111, ComfyUI, Forge) default to stacking — you load several LoRAs, assign each a weight, and the software applies them all at runtime. AUTOMATIC1111's base merger implements the weighted sum formula (1−α)A + αB that many tools reuse AUTOMATIC1111 merger. Merging is an explicit offline step using tools like kohya-ss's merge_lora.py, the Supermerger extension Civitai #1540, or PEFT's add_weighted_adapter().

How the Weight Math Differs

Single LoRA

For a base weight matrix W₀ ∈ ℝ^(d×k), a LoRA adds:

W = W₀ + (α / r) · B A

where B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), r ≪ min(d,k), and α is a training-time scaling constant. A runtime strength s further modulates the effect:

W = W₀ + s · (α / r) · B A

This is a linear addition — the base weights never change on disk. The original LoRA paper confirms the merge-at-deploy formulation with zero latency cost arXiv:2106.09685.

Stacking vs Merging

Stacking applies each adapter's update independently and sums them at runtime. The rank structure of every adapter is preserved: a rank-16 character LoRA and a rank-32 style LoRA each keep their original decomposition.

Merging collapses the adapters into one matrix pair before inference. For a weighted sum of LoRAs with ratios w_i:

W_merged = W₀ + Σᵢ [ w_i · (α_i / r_i) · B_i A_i ]

kohya-ss's merge_lora.py implements this by concatenating B matrices along the output dimension and A along the input dimension, then scaling each by alpha / dim × ratio kohya-ss merge_lora.py. Supermerger adds per-block control — different merge ratios for different U-Net blocks — but the underlying math is the same weighted summation Supermerger README.

Baking Into the Checkpoint

Folding the merged LoRA directly into the base model (W_final = W₀ + ΔW_merged) produces a standard checkpoint with zero adapter overhead. This is what Supermerger's "Merge to Checkpoint" and kohya-ss's merge_to_sd_model() do — the model is the adapted model.

When Merging Beats Stacking

VRAM and memory. Each stacked LoRA loads its matrices into GPU memory alongside the base model. Merged or baked models have one set of adapter matrices (or none). On limited VRAM with several LoRAs, merging often prevents OOM.

Inference latency. Stacking means n additional matrix multiplications per targeted layer per step. Merging means one. Over many steps, the saving compounds — especially with torch.compile.

Deployment. A single .safetensors file or baked checkpoint is trivial to share and load in production. No adapter management logic, no weight-scheduling code.

Compilation and quantization. torch.compile and quantization (bitsandbytes, AWQ, GPTQ) work best on static graphs. Stacked LoRAs introduce dynamic control flow that can trigger recompilation; a merged model is a plain nn.Module.

The Quality Cost of Collapsing Adapters

Why Merging Can Degrade Results

Each LoRA is trained independently. When you merge with a simple weighted sum, you assume the optimal combined adaptation is a linear combination of the individual optima. Neither this nor the non-interference of rank subspaces is guaranteed. Empirically, merged LoRAs sometimes blur distinct concepts — a character LoRA and a style LoRA merged together may produce a character that almost looks right but lacks the style's crisp signatures.

Supermerger's documentation acknowledges that the "SAME TO STRENGTH" option (taking √ratio during merge) is needed to make the merged result match runtime behavior — the naive weighted sum does not preserve per-adapter scaling semantics Supermerger README.

When Stacking Preserves Quality Better

Stacking applies each adapter at its native, trained strength, modulated only by the user's runtime weight. Each adapter's internal geometry is untouched, so clean subspaces for different concepts can coexist without the averaging that merging imposes.

Practical Guideline

Tools Comparison

TIES and DARE (PEFT) prune small weights and resolve sign conflicts before averaging. They can apply to LoRA matrices, but adapter-specific evidence is thin — treat as experimental.

Common Errors and Fixes

FAQ

When does merging produce better results than stacking? Merging doesn't inherently produce better quality — it trades quality for efficiency. It wins when VRAM, speed, or deployment constraints matter more than per-adapter fidelity.

How does merging affect quality compared to stacking? Merging averages the low-rank updates, which can blur concept boundaries. Stacking preserves each adapter's geometry but adds inference overhead. The gap is largest when adapters target the same layers.

How do I choose between them? Start with stacking. If you hit OOM, latency limits, or deployment constraints, merge with kohya-ss or Supermerger. Test both side-by-side — the difference is case-dependent.

Further Reading

• How LoRAs Work — mechanism, strength, stacking, and matching to base models
• Model Merge Tools — full merge tool comparison including DARE, TIES, and block weights
• LoRA Library Management — organizing, versioning, and sharing adapters

Sources

1. LoRA paper (arXiv:2106.09685) — Original formulation: frozen weights + low-rank updates, merge-at-deploy with zero latency cost. https://arxiv.org/abs/2106.09685
2. AUTOMATIC1111 ui_checkpoint_merger.py — Weighted sum merge: (1−α)A + αB. https://raw.githubusercontent.com/AUTOMATIC1111/stable-diffusion-webui/master/modules/ui_checkpoint_merger.py
3. Hugging Face Diffusers "Merge LoRAs" — set_adapters() for stacking, add_weighted_adapter() with TIES/DARE for merging. https://huggingface.co/docs/diffusers/main/en/using-diffusers/merge_loras
4. kohya-ss merge_lora.py — LoRA-to-LoRA merging (weighted sum, concat) and LoRA-to-checkpoint baking. https://raw.githubusercontent.com/kohya-ss/sd-scripts/main/networks/merge_lora.py
5. Supermerger README — Block-weighted merging, "Merge to Checkpoint," "SAME TO STRENGTH" scaling. https://raw.githubusercontent.com/hako-mikan/sd-webui-supermerger/main/README.md
6. PEFT LoRA conceptual guide — Rank-stabilized scaling, merge_and_unload(), multi-adapter combining. https://huggingface.co/docs/peft/main/en/conceptual_guides/lora
7. Civitai article #1540 (freek22) — Community report on Supermerger merging. https://civitai.com/articles/1540