IPAdapter vs ControlNet vs Regional Prompter in ComfyUI

TL;DR

• IPAdapter transfers a subject or style from a reference image; it adds image-prompt cross-attention on top of the frozen text cross-attention (tencent-ailab/IP-Adapter).
• ControlNet dictates spatial structure (edges, depth, pose, segmentation); it trains a copy of UNet encoder blocks fed by a control map (lllyasviel/ControlNet).
• Regional Prompter binds different text prompts to different image regions, which fixes attribute bleed between characters (sd-webui-regional-prompter).

The correction you need before anything else

A widely circulated 2024 ComfyUI tutorial tells you to tune an IPAdapter noise slider (0.01 = "adds noise", 1.0 = "removes noise"). That parameter no longer exists. The old IPAdapter Apply node was removed and replaced by IPAdapter Advanced, with image_negative as the current negative-conditioning input — an image, not a 0.01–1.0 slider. Without one, the node defaults to a zero image: image_negative = image_negative if image_negative is not None else torch.zeros([1, clipvision_size, clipvision_size, 3]). So the old empty-image-negative idea survives, just as an image input. Ignore any tutorial quoting 0.01/1.0 numbers.

What is IPAdapter and when does it win?

IPAdapter is a roughly 22-million-parameter adapter that gives a pretrained text-to-image diffusion model an image prompt on top of its text prompt. The original paper repo describes "an effective and lightweight adapter to achieve image prompt capability for the pre-trained text-to-image diffusion models. An IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fine-tuned image prompt model." The ComfyUI maintainer calls it "a 1-image lora" (cubiq README) — and when one image is not enough, that is the point where training an actual LoRA starts paying off. It wins for appearance transfer — subject, face, or style from a reference — and loses when you need pixel-accurate layout, because it carries no structural map.

Key settings: start weight/scale at 0.8 for the default linear weight_type; use 0.5 for balanced text-plus-image and 1.0 for image-only. Always use a square reference image — CLIP's default processor center-crops, so non-square inputs "will miss the information outside the center" (tencent README).

IPAdapter model variants at a glance

FaceID models are a separate family: they "require insightface" and "most FaceID models also need a LoRA," with the warning that "each FaceID model has to be paired with its own specific LoRA" (cubiq README).

How do I make IPAdapter only affect part of the image?

The IPAdapter Advanced node has an attn_mask input that restricts where the reference applies. Per the node docs: "The mask will define the area of influence of the IPAdapter models on the final image. Black zones won't be affected, white zones will get maximum influence. It can be a grayscale mask." The mask should match the latent's size or aspect ratio. The diffusers docs describe the same mechanism independently: "Binary masking enables assigning an IP-Adapter image to a specific area of the output image, making it useful for composing multiple IP-Adapter images. Each IP-Adapter image requires a binary mask."

This is the modern, supported way to compose several reference images into one frame. The cubiq regional conditioning example demonstrates it with a MaskFromRGBCMYBW+ node splitting a color-coded mask into per-region masks that each feed an IPAdapter param block. (An older tutorial cites examples/IPAdapter_2_masks.json, a filename no longer in the examples directory — the current equivalent is ipadapter_regional_conditioning.json.)

Two further IPAdapter Advanced inputs matter for regional work: weight_type ("style transfer (SDXL) only works with SDXL and it's a very powerful tool to transfer only the style of an image but not its content") and start_at/end_at timestepping ("if you start later (eg: start_at=0.3) the generated image will have a very light conditioning") — both from the node docs.

How do I transfer only style, not content, with IPAdapter?

Use InstantStyle. Its two-part method is documented in the project README: subtract content text features from the image features so "the style and content can be explicitly decoupled," then inject only into the style-handling attention blocks — "up blocks.0.attentions.1 and down blocks.2.attentions.1 capture style (color, material, atmosphere) and spatial layout (structure, composition) respectively." It is "supported in ComfyUI_IPAdapter_plus developed by our co-author," and the weight_type: style transfer (SDXL) option exposes the same idea.

What is ControlNet and when does it win?

ControlNet adds a spatial condition — Canny edges, depth maps, OpenPose skeletons, segmentation, scribbles, etc. — through a trainable copy of UNet encoder blocks, while a "locked" copy preserves the base model. A "zero convolution" (a 1×1 conv with zero weight and bias) means it adds no distortion before training: "before training, all zero convolutions output zeros, and ControlNet will not cause any distortion" (ControlNet README). The maintainer's FAQ explains why zero weights still learn: "as long as $x \neq 0$, one gradient descent iteration will make $w$ non-zero."

ControlNet wins when you need structure — pose, edges, depth, layout. The ComfyUI ControlNetApplyAdvanced node exposes strength (0.0–10.0, default 1.0), start_percent and end_percent (both 0–1) (ComfyUI nodes.py), matching IPAdapter's timestepping idiom. Multiple ControlNets compose: "more than one ControlNet can be easily composed to multi-condition control" (ControlNet README). For matching ControlNet types to tasks, see ControlNet models for composition control.

Regional Prompter: prompts locked to regions

Regional Prompter is an A1111 extension that lets "different prompts... be specified for different regions," separated by the BREAK keyword. ComfyUI users hit the same need when IPAdapter masks feel heavy-handed and a text-only split suffices. The syntax:

a man and a woman, a man with black hair
BREAK
a man and a woman, a woman with blonde hair

The first block applies to the left region, the second to the right. The shared "a man and a woman" matters: without it the model sees two single-person prompts and renders one person. The ADDCOMM keyword automates this by prepending the first block to every region, and ADDBASE plus a base ratio blends a base prompt by weight rather than concatenating it — "common clause combines prompts, and base clause combines weights (like img2img denoising strength)" (hako README).

Common prompt vs base prompt

Attention vs Latent mode in Regional Prompter

Regional Prompter has two calculation modes with a real accuracy/speed trade-off. Per the README, "In the case of a 512 x 512 image, Attention mode reduces the size of the region to about 8 x 8 pixels deep in the U-Net, so that small areas get mixed up; Latent mode calculates 64*64, so that the region is exact." Latent mode is slower — "the generation time is the number of areas x the generation time of one pic" — but it is the mode that "allows separating LoRAs to some extent." Internally Attention uses BREAK and Latent uses AND, auto-converted so you can type either.

ADetailer tokens: [SEP], [SKIP], [PROMPT]

When ADetailer detects several faces, three tokens control per-face prompts. If you have not set ADetailer up yet, start with our guide to fixing faces and hands automatically. From the wiki:

• [SEP] splits one prompt into per-object prompts, "split by the regex \s*[SEP]\s*," applied in detection order — which is "highly arbitrary, so use with caution."
• [SKIP] skips inpainting for that part (regex ^\s*[SKIP]\s*$).
• [PROMPT] is replaced by the original input prompt.

Pairing Regional Prompter for placement with ADetailer for face repair is the standard multi-character recipe. An often-cited claim that face processing order is adjustable via a settings toggle is not supported by the ADetailer wiki — detection order is arbitrary.

Common errors and fixes

How to choose: a decision table

FAQ

How do I make IPAdapter only affect part of the image? Connect a mask to the attn_mask input on IPAdapter Advanced. White areas get full reference influence, black areas get none, and grayscale is allowed. The mask should match the latent's size or aspect ratio. The diffusers docs call this "binary masking" and require one binary mask per IP-Adapter image.

What's the difference between Regional Prompter common prompt and base prompt? A common prompt (ADDCOMM) is prepended to every region's prompt — concatenation. A base prompt (ADDBASE) is blended by a base ratio weight, like img2img denoising strength: a ratio of 0.2 means 20% base plus 80% region. Use common when the scene framing must repeat; use base when that framing is too strong and you want to dial it down (hako README).

Why do my LoRAs corrupt when using Regional Prompter? It is a known issue in Latent mode — the README lists mitigations but states "no solution as of yet." Reduce CFG, reduce the LoRA weight, increase sampling steps, or use the negative textencoder and negative U-net parameters (0–1 per LoRA). Inpainting is the last resort. The deep cause is not documented; do not assume one.

Can I use IPAdapter and ControlNet together? Yes. Tencent ships dedicated ip_adapter_controlnet_demo notebooks for "structural generation with image prompt," and ControlNets are documented as composable. A common setup uses ControlNet (depth or OpenPose) for layout and IPAdapter for the subject's appearance, each with its own strength/weight and start/end percentages.

How do I transfer only style, not content, with IPAdapter? Use InstantStyle, which subtracts content text features from the image features and injects only into the style attention blocks (up_blocks.0.attentions.1 for style, down_blocks.2.attentions.1 for layout). It is built into ComfyUI_IPAdapter_plus; on SDXL you can also try weight_type: style transfer (SDXL).

What ADetailer tokens separate faces? [SEP] splits one prompt into per-object prompts in detection order (which the wiki calls "highly arbitrary"), [SKIP] skips inpainting for that part, and [PROMPT] is replaced by the original input prompt. ADetailer itself runs in three steps: generate, detect-and-mask, then inpaint (README).

What IPAdapter weight should I start with? The cubiq README suggests lowering weight "to at least 0.8" and increasing steps; the NODES.md calls 0.8 "a good starting point" for the default linear weight type. For balanced text-plus-image conditioning Tencent and the diffusers docs both point to scale=0.5, reserving 1.0 for image-only generation.

Does Regional Prompter work 100% of the time? No. One community tutorial reports roughly a 75% success rate; treat that as community-reported rather than a measured figure (stable-diffusion-art.com). Attribute bleed between characters still happens, which is why ADetailer face repair is usually paired with it.

Sources

• cubiq/ComfyUI_IPAdapter_plus — main extension repo and README.
• NODES.md — IPAdapter Advanced inputs (weight, attn_mask, image_negative, weight_type, start_at/end_at).
• cubiq README — model variant strengths, weight ≥ 0.8, "1-image lora," FaceID requirements.
• IPAdapterPlus.py — image_negative default zero tensor.
• Issue #108 — IPAdapter Apply → IPAdapter Advanced deprecation; CLIP encoder and dtype fixes.
• ipadapter_regional_conditioning.json — mask-based regional IPAdapter example.
• tencent-ailab/IP-Adapter — original paper repo; 22M params; square-image note; scale 1.0 vs 0.5; ControlNet demos.
• sd-webui-regional-prompter — BREAK/ADDCOMM/ADDBASE; Attention vs Latent; LoRA corruption known issue.
• Bing-su/adetailer — ADetailer 3-step pipeline (generate → detect → inpaint).
• ADetailer wiki/Advanced — [SEP], [SKIP], [PROMPT]; detection order is arbitrary.
• lllyasviel/ControlNet — locked + trainable copies, zero convolution, composable ControlNets.
• ControlNet FAQ — why zero conv learns (gradient non-zero when x ≠ 0).
• ComfyUI nodes.py — ControlNetApplyAdvanced: strength 0–10 default 1.0, start_percent/end_percent 0–1.
• InstantStyle — feature subtraction + style-block-only injection; native support in cubiq IPAdapter_plus.
• diffusers IP-Adapter docs — scale 1.0 image-only, 0.5 balanced; binary masking.
• stable-diffusion-art.com regional prompter — common prompt necessity; ~75% success (community-reported); attribute bleed.