Create an Inpainting Checkpoint from Any Model

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

TL;DR

• Merge recipe: In Automatic1111's Checkpoint Merger, select Add Difference mode with Model A = inpainting base, Model B = your fine-tune, Model C = matching base model. Multiplier is ignored for this mode.
• Architecture difference: Inpainting UNets have 9 input channels (4 latent + 4 masked-image latent + 1 mask) vs. 4 for base models. The extra channels are zero-initialized during inpainting fine-tuning.
• Seed/noise control: For deterministic edits, pass both a seeded torch.Generator and explicit latents tensor to the pipeline; strength < 1.0 preserves some original latent signal.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

Why Convert a Checkpoint Instead of Downloading One?

Dedicated inpainting checkpoints exist for SD 1.5 and SDXL, but they're tied to the base model they were trained from. If you've fine-tuned a model on a specific style, character, or domain, the official inpainting weights won't carry those adaptations. The Add Difference merge (A + (B - C)) grafts the inpainting capability onto your checkpoint while preserving its learned concepts.

This technique works because the inpainting delta — the weights that changed when the base model was fine-tuned for inpainting — is isolated by subtracting the base (C) from the inpainting base (A). Applying that delta to your fine-tune (B) transfers only the inpainting-specific adaptation.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

The Architecture Difference: 9 Channels vs. 4

A standard Stable Diffusion UNet expects 4-channel latents from the VAE encoder. An inpainting UNet expects 9 channels:

The 5 additional channels are zero-initialized when inpainting training begins, so the model learns to condition on the masked region and mask from scratch. This is why a regular checkpoint simply fails at inpainting — its UNet has no weights for those extra inputs.

Source: The SD 1.5 inpainting model card states: "For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint."

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

SD 1.5 Merge Recipe

The output file appears as <YourModelName>.inpainting.safetensors in your models folder.

Note: The benjamin-paine repository mirrors the official CompVis/Stability weights. Use the official model card as the canonical source.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

SDXL Merge Recipe

Critical: Model C must be the FP16 VAE version. The Diffusers team trained XL Inpainting with the FP16 VAE fused in; using a different base (fp32 VAE or no VAE) applies a mismatched delta and yields a broken VAE.

Source: Benjamin Paine's model card: "You must specifically use the two files present in this repository... using a different XL base will result in an incorrect delta being applied to the inpainting checkpoint, and the resulting VAE will be nonsensical."

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

How Add Difference Works (The Mechanism)

The formula theta_A + (theta_B - theta_C) computes:

1. theta_A - theta_C = the inpainting delta (weights added/changed when the base model was fine-tuned for inpainting)
2. theta_B + delta = your fine-tune with the inpainting adaptation grafted on

Because A and C share the same architecture (and for SDXL, the same VAE), the subtraction cancels out the shared base weights, leaving only the inpainting-specific changes: the 9-channel input projection, attention modifications, and any VAE statistics baked into the delta.

Source: A1111 merge_models.py implements add_difference as theta_0 + (theta_1 - theta_2) with no multiplier scaling.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

Common Error: BigLust v1.0 Metadata Crash

Source: Civitai article 833 workaround.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

Controlling Seed and Noise for Repeatable Edits

Three levers determine determinism in the StableDiffusionInpaintPipeline:

For exact reproducibility across runs, supply both a seeded generator and explicit latents.

Source: diffusers StableDiffusionInpaintPipeline.__call__ signature documents generator ("make generation deterministic") and latents ("Pre-generated noisy latents to be used as inputs").

Denoising Strength vs. Mask Interaction

The strength parameter controls how much noise is added to the input image before denoising the masked region:

• 1.0 → masked region = pure noise → full regeneration guided only by prompt
• 0.3–0.5 → retains more original content → finer edits, better structure preservation
• 0.7 → common starting point for "rough sketch → coherent image" (heuristic, not a rule)

Source: Pipeline docs: "A value of 1 essentially ignores image." Community workflow (Civitai 4451) recommends 0.7 for first pass, 0.3 for detail refinement.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

Workflow: From Sketch to Inpaint-Ready Result

1. Prepare the inpainting checkpoint using the merge recipe above.
2. Open an img2img/inpaint interface (A1111, ComfyUI, or Krita AI Diffusion).
3. Draw a crude mask over the region to regenerate; set strength ≈ 0.7.
4. Prompt the desired content for the masked area.
5. Iterate: lower strength to 0.3–0.4, refine mask/prompt, re-run.
6. For exact reproducibility: capture the latents tensor from a good run, then re-use it with the same seed.

Tooling: Krita AI Diffusion (ComfyUI) provides a fast sketch → inpaint loop. ControlNet's "inpaint only" processor handles outpainting. ADetailer automates face/body touch-ups.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

Quick Reference: SD 1.5 vs. SDXL Merge Settings

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

FAQ

Can I use a regular checkpoint for inpainting without merging?

No. The UNet lacks the 5 additional input channels (4 masked-image latent + 1 mask). It will either error or produce garbage.

Does the Multiplier setting matter for Add Difference?

No. The A1111 implementation ignores multiplier for this mode; the formula is fixed as A + (B - C).

Why must SDXL use the FP16 VAE as Model C?

The XL inpainting model was trained with the FP16 VAE fused in. The delta A - C contains VAE-specific statistics. A mismatched base applies the wrong delta and breaks the VAE.

How do I get the exact same inpaint result twice?

Pass the same seeded torch.Generator and the same latents tensor to the pipeline. strength must also match.

What does strength=0.5 actually do?

It adds noise halfway to pure noise. At 0.5, the masked region is noised halfway toward pure noise; the denoiser preserves more structure while still allowing prompt-guided changes.

Can I merge a LoRA into the inpainting checkpoint instead?

Yes — load the merged inpainting checkpoint, then apply LoRAs at inference time. The merge recipe only handles the base checkpoint architecture.

Where do I find the official SD 1.5 inpainting weights?

stable-diffusion-v1-5/stable-diffusion-inpainting on HuggingFace (CompVis/Stability). The benjamin-paine repo is a mirror.

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial

Sources

Related reads: Stable Diffusion checkpoint training guide · ControlNet inpainting and outpainting · ComfyUI inpainting workflow tutorial