Sketch to Image with Scribble and Lineart ControlNet

TL;DR

• Use Scribble ControlNet for freehand sketches and Lineart ControlNet for pen-and-ink outlines; both work best when the drawing is the primary input and the prompt supplements rather than replaces it.
• Keep the ControlNet weight between 0.5 and 0.95; 0.6 is a common starting point for sketch-to-image pipelines.
• A denoise budget of 0.3 at the final pass preserves composition while adding detail — lower values protect structure, higher values add texture at the cost of fidelity.

What Kind of Drawing Do You Need?

Scribble ControlNet is designed for freehand sketches — rough shapes, stick figures, and loose outlines. It does not require clean linework or precise proportions. The ComfyUI examples page provides a Scribble ControlNet demo using the AnythingV3 model and a simple pencil sketch as input.

Lineart ControlNet is a separate model intended for cleaner, more defined strokes — ink drawings, architectural sketches, and anime lineart. RunComfy's ComfyUI guide lists the Lineart variant as suitable for "clean line drawings" alongside the Scribble variants designed to handle rougher input source.

The PainterNode in ComfyUI lets you draw directly in the node interface and pipe the result into ControlNet, with support for MyPaint brushes and symmetry tools. For a full ComfyUI setup guide, see Getting Started with ComfyUI.

How Much Detail Does Your Drawing Need?

There is no published quantitative threshold for how much detail a ControlNet sketch must contain, but practice from the drawing-to-image community indicates that key structural elements should be present: recognizable silhouettes, major proportions, and a clear subject boundary. Fine details such as facial features, text, or small objects are better left to the model's generative capability rather than drawn explicitly.

A rough pencil sketch with the main shapes and posture is often enough for Scribble ControlNet to produce a coherent result. Adding more detail increases the model's reliance on your drawing and reduces its freedom to interpret the prompt. If you find the output ignores your prompt, the drawing may be too detailed; if the output loses structure, the drawing may be too vague.

Scribble ControlNet expects simplified line drawings as input. The ComfyUI examples page notes that ControlNet models require preprocessed images in a specific format — the ControlNetApply node does not convert regular photographs automatically; you must preprocess them separately source.

The Denoise Budget: Keeping Your Composition

In the XL6 Hephaistos "high res fix" workflow, the second generation pass uses a denoising value of 0.3 — described as works "like a high res fix." The guidance is to lower it when the result alters the image too much, and raise it when you want more detail source.

The LCM acceleration model — used in the drawing2img workflow with LCM_LoRA_Weights_SD15.safetensors — reduces inference steps to 2–8 for SD 1.5 models source, making it practical to iterate quickly when dialing in the denoise budget. The recommended guidance_scale is between 1.0 and 2.0, or disabled entirely source. Learn more about LCM LoRA acceleration.

Workflow Comparison

See the Sources section for full URLs and per-column references.

Common Errors and Fixes

Sketch looks ignored by the model. The drawing may be too vague — ControlNet needs clear stroke structure to guide generation. Try increasing line weight or adding more defining contours source.

Output composition drifts from the sketch. This usually means the denoise value is too high in the final pass. Lower it to 0.2 or below to tighten composition fidelity, or raise it to 0.4–0.5 if you want more detail and are willing to accept structural shifts. The XL6 Hephaistos author recommends 0.3 as a balanced starting point source.

Out-of-memory errors with Stable Cascade. Use the bf16 or Lite variants — Stage B bf16 + Stage C bf16 runs at ~12 GB, and the Lite combo at ~5 GB source. The full-precision Stage B + Stage C combination requires ~22 GB source.

LCM acceleration not triggering. Ensure the scheduler is changed to LCMScheduler and guidance_scale is between 1.0 and 2.0 (or set to 0 for guidance-free generation) source.

FAQ

What is the difference between Scribble and Lineart ControlNet?

Scribble ControlNet is trained on rough freehand sketches and tolerates imprecise lines. Lineart ControlNet expects clean, defined strokes and produces sharper structural adherence. Use Scribble for rough concept sketches and Lineart for refined drawings source. For a deeper dive, see ControlNet Types Explained.

Is LCM-LoRA compatible with all SD models?

LCM-LoRA comes in model-specific variants: an SD 1.5 version (67.5M parameters) and an SDXL version (197M parameters). You must match the LoRA to your base model. The drawing2img workflow uses the SD 1.5 variant source, source.

Can I use this workflow on an 8 GB GPU?

The Stable Cascade lite variant (Stage B Lite + Stage C Lite) runs at ~8 GB. The SD 1.5 LCM workflow requires less VRAM. For any GPU with at least 6 GB of VRAM, the SD 1.5 LCM workflow with LCM acceleration at 512×512 can run comfortably source.

What ControlNet weight should I start with?

The drawing2img workflow sets it to 0.6 and the associated Note node suggests the range 0.5–0.95 source. Start at 0.5 and increase if the output is too loose; decrease if the sketch dominates the result.

Does Stable Cascade support ControlNet?

Stable Cascade's model card on HuggingFace lists "ControlNet" among its supported extensions alongside LoRA training and finetuning source.

How do I preserve my sketch's composition through upscaling?

The XL6 Hephaistos workflow uses a two-pass approach: generate at base resolution, upscale 4×, downscale 50%, then run a second pass at low denoising (0.3). The downscale step re-anchors the composition before the final refinement pass source.

Sources

• https://raw.githubusercontent.com/ecjojo/ComfyUI-Templates/main/Civitai_Workflow/drawing2img.json — drawing2img workflow JSON: ControlNetApply strength 0.6, Note node "0.5 ~ 0.95", PainterNode input, LCM LoRA SD15
• https://civitai.com/articles/1648 — XL6 Hephaistos workflow: refiner steps, denoising 0.3, high-res fix pipeline
• https://civitai.com/articles/4161 — Stable Cascade VRAM requirements, model sizes, ControlNet support
• https://huggingface.co/stabilityai/stable-cascade — Stable Cascade architecture, compression factor 42, ControlNet compatibility
• https://huggingface.co/latent-consistency/lcm-lora-sdxl — LCM-LoRA SDXL: 2–8 steps, guidance_scale 1.0–2.0, ControlNet example with conditioning_scale=0.5
• https://huggingface.co/latent-consistency/lcm-lora-sdv1-5 — LCM-LoRA SD 1.5: 2–8 steps, ControlNet example with conditioning_scale=0.8
• https://comfyanonymous.github.io/ComfyUI_examples/controlnet/ — Scribble ControlNet example, preprocessing requirements, T2I-Adapter efficiency
• https://raw.githubusercontent.com/AlekPet/ComfyUI_Custom_Nodes_AlekPet/master/PainterNode/README.md — PainterNode features for sketch input