Stable diffusion empty prompt inpainting: A1111 vs ComfyUI and when prompts hurt edits

TL;DR

• IP-Adapter with scale=1.0 and text_prompt="" works in both A1111 and ComfyUI for clean inpainting
• ControlNet inpaint_only+lama preprocessors provide better border alignment than text-only guidance in A1111 (source)
• High prompt weights can degrade results; optimal scale is 0.5 for combined image plus text guidance (source)

Core Mechanism

An empty prompt generates the unconditional embedding, which carries no directional information. The model reverts to its unconditional distribution, which is why promptless inpainting without image conditioning produces random output that does not fit the existing image (source).

IP-Adapter provides the missing image signal. It uses a CLIP image encoder to extract features from the surrounding image, then projects those features into the UNet's cross-attention space via a lightweight adapter. The adapter contains only 22M parameters yet can match or exceed a fine-tuned image prompt model for consistency (source). With scale=1.0 and text_prompt="", the adapter becomes the sole guidance signal (source).

For broader inpainting setup context, see ComfyUI inpainting workflows. For a full breakdown of IPAdapter settings, weight types and regional masking, see IPAdapter vs ControlNet vs Regional Prompter.

A1111 vs ComfyUI: Technical Comparison

The table above shows that A1111 provides a simpler path to promptless inpainting with its ControlNet extension, while ComfyUI offers more granular node-level control and supports regional prompting that A1111 lacks. If you are troubleshooting ControlNet-related setups, A1111 Forge repair and install may be relevant source source.

Decision: When to Use Empty Prompts vs Text Prompts

Common Errors and Fixes

Error: IP-Adapter Loads Models Twice

Symptom: Memory waste and duplicate computation when using multiple IP-Adapter nodes.

Cause: Daisy-chaining not done correctly; first Unified Loader's ipadapter input was connected.

Fix: Leave the first loader's ipadapter input disconnected; chain subsequent loaders through the ipadapter in/out ports.

Verifiable quote: "Multiple unified loaders should always be daisy chained through the ipadapter in/out. Failing to do so will cause all models to be loaded twice. For the first unified loader the ipadapter input should never be connected." (source)

Error: Non-Square Images Produce Poor Results

Symptom: Missing edge information on the left/right or top/bottom of generated content outside the center square.

Cause: CLIP center-crops input images to 224x224, discarding data outside the central crop region (source).

Fix: Resize images to square before feeding to IP-Adapter, or accept the center-crop limitation.

Verifiable quote: "As the image is center cropped in the default image processor of CLIP, IP-Adapter works best for square images. But you can just resize to 224x224 for non-square images." (source)

Error: High Weights Burn the Image

Symptom: Visual artifacts, washing out of image details, or blown-out colors at high IP-Adapter weight settings.

Cause: Default embeds_scaling parameter produces artifacts at weights above 1.0 (source).

Fix: Use "K+mean(V) w/ C penalty" embeds_scaling for higher weights.

Verifiable quote: "This parameter has a small impact on how the model reacts to text prompting. K+mean(V) w/ C penalty grants good quality at high weights (>1.0) without burning the image." (source)

FAQ

What happens when you use an empty prompt in stable diffusion inpainting?

An empty prompt generates the unconditional embedding, which carries no directional information. The model reverts to its unconditional distribution, producing random content that does not fit into the existing image. This is why vanilla Stable Diffusion "does not really work well without a text prompt" (source).

See also: the Stable Diffusion Inpainting model card confirms the model was trained with classifier-free guidance sampling that drops text conditioning 10% of the time, meaning it was built to rely on text as its primary conditioning signal (source).

How does IP-Adapter inpainting work without a text prompt?

IP-Adapter uses a CLIP image encoder to extract features from the existing image, then projects those into the UNet's cross-attention space via a 22M-parameter adapter (source). With scale=1.0 and text_prompt="", the adapter becomes the sole guidance signal. The UNet's cross-attention receives only image-based features, which carry enough semantic information to generate content matching the image's style, color palette, and composition. For a full walkthrough of IPAdapter settings in ComfyUI, see IPAdapter vs ControlNet vs Regional Prompter.

Why is my inpainting result random without a prompt?

Random results mean you are using vanilla Stable Diffusion without image conditioning. The SD Inpainting model was trained with 440k inpainting steps at 512x512 resolution on top of its base 595k training steps, but it still relies on text conditioning as its primary signal (source).

Add image-based guidance using either ControlNet inpaint_only+lama preprocessor for border alignment, or IP-Adapter with scale=1.0 and text_prompt="" for style preservation (source).

What is the difference between inpaint_only and inpaint_only+lama preprocessor?

Both are ControlNet preprocessing options available in A1111's sd-webui-controlnet extension and via comfyui_controlnet_aux in ComfyUI (source):

• inpaint_only: Basic preprocessing using ControlNet's inpaint hint generation
• inpaint_only+lama: Uses LaMa inpainting algorithm to pre-fill the masked area with plausible content before feeding it as a hint to ControlNet

The +lama variant typically produces better results for challenging inpainting scenarios but at higher computational cost.

When does adding a text prompt hurt inpainting quality?

The IP-Adapter documentation notes that for image-only guidance, set scale=1.0 and text_prompt="" or generic phrases like "best quality." For combined image plus text guidance, setting scale=0.5 typically yields good results (source). Higher scales can reduce consistency with the image prompt because the model tries to balance conflicting signal directions.

How to do promptless inpainting in ComfyUI vs Automatic1111?

Automatic1111 (A1111):

• Uses sd-webui-controlnet extension (source)
• ControlNet inpaint_only+lama with "ControlNet is more important" option (source)
• Simpler UI but limited advanced options

ComfyUI:

• Uses comfyui_controlnet_aux plus ComfyUI_IPAdapter_plus (source)
• IP-Adapter's inpaint input on Apply IP Adapter node for combined image plus ControlNet guidance (source)
• Support for Regional Prompting workflow via ProfessorLich's Get/Set nodes (source)

ComfyUI offers more granular control but requires understanding the node graph. A1111 provides easier access for basic promptless inpainting.

Sources

1. IP-Adapter official repository and documentation - https://github.com/tencent-ailab/IP-Adapter
2. ComfyUI_IPAdapter_plus - https://github.com/cubiq/ComfyUI_IPAdapter_plus
3. ComfyUI_IPAdapter_plus nodes documentation - https://github.com/cubiq/ComfyUI_IPAdapter_plus/blob/main/NODES.md
4. ComfyUI ControlNet Auxiliary Preprocessors - https://github.com/Fannovel16/comfyui_controlnet_aux
5. sd-webui-controlnet discussion #1464 - https://github.com/Mikubill/sd-webui-controlnet/discussions/1464
6. Stable Diffusion Inpainting model card - https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting
7. Civitai article 1907 (Acly) - https://civitai.com/articles/1907
8. Civitai article 9534 (ProfessorLich) - https://civitai.com/articles/9534