Latent vs Model Upscaling: Complete AI Image Guide

TL;DR

• Latent upscaling preserves composition better for photorealistic work but runs slower (~280s on RTX 4090 for Flux) [https://civitai.com/articles/7181].
• Model upscalers like RealSR DF2K pair well with tiled diffusion [https://openmodeldb.info/models/4x-realsr-df2k].
• Tiled workflows (MultiDiffusion) let you upscale beyond VRAM limits by processing images in overlapping tiles [https://civitai.com/articles/7036].
• Use hires fix for quick 2× boosts; switch to latent upscaling when composition fidelity matters most [https://comfyanonymous.github.io/ComfyUI_examples/2_pass_txt2img/].

Latent Upscaling vs Model Upscalers: Core Difference

Latent upscaling operates inside the diffusion model's compressed representation. The model generates higher-resolution detail while keeping the overall composition locked to the original latent structure. Model upscalers (RealSR, ESRGAN variants) work in pixel space — they take a rendered image and reconstruct detail using learned priors from training data.

The Flux latent upscaler workflow demonstrates this principle: a low-resolution first pass establishes composition, then a second pass upscales in latent space to roughly 2× the original dimensions [https://civitai.com/articles/7181]. Some compositional drift occurs at the doubling point, but the overall structure holds.

Hires Fix: The Built-In Shortcut

Hires fix is not a separate algorithm — it's a workflow pattern. Generate at low resolution, upscale (latent or pixel), then run a second img2img pass at low denoising strength. In ComfyUI, txt2img and img2img use the same sampler node; hires fix simply feeds an upscaled latent back into the sampler with reduced denoise.

ComfyUI's documented example shows two variants [https://comfyanonymous.github.io/ComfyUI_examples/2_pass_txt2img/]:

• Latent hires fix: Upscale in latent space, stay in latent space
• Pixel hires fix: Decode to pixels, apply ESRGAN/RealSR, re-encode to latent

The pixel variant introduces a VAE encode/decode round-trip, which can soften fine detail but gives access to powerful pixel-space models like RealSR DF2K.

Tiled Workflows: MultiDiffusion for Large Images

When target resolution exceeds VRAM, tiled diffusion splits the image into overlapping windows, processes each independently, then blends the overlaps. MultiDiffusion formalized this as an optimization problem binding multiple diffusion paths with shared constraints.

In ReForge (a Forge fork of Automatic1111), the documented workflow is [https://civitai.com/articles/7036]:

This processes each tile at the model's native resolution (e.g., 832×1216), avoiding the quality loss of native high-res generation while staying within VRAM limits.

Photorealistic vs Illustration: Which Method Works

Photorealistic — Favor Latent Upscaling

Photorealistic images rely on coherent lighting, consistent textures, and plausible geometry. Latent upscaling preserves these because the diffusion model's internal representation already encodes 3D structure. The Flux workflow's seed modulation (third pass at ~112s on RTX 4090) lets you reroll fine detail while locking the base composition — useful for fixing hallucinated fingers or texture anomalies without restarting [https://civitai.com/articles/7181].

Model upscalers trained on photo data (RealSR DF2K won NTIRE 2020 Real-World Super-Resolution) excel at restoring edge acuity and suppressing compression artifacts. They work best as a pre-step before latent refinement [https://openmodeldb.info/models/4x-realsr-df2k].

Illustration / Anime — Favor Model Upscalers + Tiled Diffusion

Illustrated content has flat color regions, hard edges, and stylized shading. Pixel-space models like RealSR DF2K handle clean line preservation better than latent diffusion, which can soften cel-shaded boundaries. The guy90 workflow demonstrates this: base generation at 832×1216, 2× RealSR upscale, then tiled img2img at 0.2 denoise with MultiDiffusion [https://civitai.com/articles/7036].

Common Errors and Fixes

FAQ

What is latent upscaling in Stable Diffusion?

Latent upscaling increases resolution inside the model's compressed latent space instead of pixel space. The diffusion model generates detail consistent with its learned prior, preserving global composition better than pixel interpolation.

How does hires fix differ from tiled upscaling?

Hires fix is a two-pass workflow at fixed 2× scale. Tiled upscaling processes arbitrary resolutions by splitting into overlapping tiles, enabling 4×, 8×, or larger outputs within the same VRAM budget.

Which upscaler model is best for anime?

RealSR DF2K is widely used for illustrated content. It won NTIRE 2020 for real-world super-resolution and the guy90 workflow uses it for anime generation [https://civitai.com/articles/7036].

Can I combine latent and model upscaling?

Yes. A common pipeline: generate at base resolution, latent upscale 1.5×, decode, pixel upscale 2× with RealSR DF2K, then optional tiled img2img refinement at 0.2 denoise.

Why does Flux latent upscaling take ~280 seconds on RTX 4090?

Flux runs diffusion steps at two resolutions sequentially. The Hyper Flux LoRA variant reduces this to ~130s by using distilled LoRAs and a faster text encoder, with slight quality trade-off [https://civitai.com/articles/7181].

What tile overlap prevents seam artifacts?

64px is the documented baseline for 832×1216 tiles in the MultiDiffusion workflow [https://civitai.com/articles/7036].

Does ADetailer work with Flux?

ADetailer uses YOLO detection models (face_yolov8n.pt at 0.660 mAP50) which are model-agnostic — they detect faces in the rendered pixel output. It works with any backend that produces standard image output, including Flux via ComfyUI or ReForge https://github.com/Bing-su/adetailer.

Sources

1. Civitai Article 7181 — Flux Latent Upscaler — workflow, performance numbers, Hyper Flux variant
2. Civitai Article 7036 — Generation Guide — ReForge workflow, MultiDiffusion parameters, RealSR DF2K usage
3. ADetailer Repository — detection model specs, installation
4. OpenModelDB RealSR DF2K — model card, NTIRE 2020 winner
5. ComfyUI Hires Fix Examples — latent vs pixel hires fix workflows
6. Bingsu/adetailer Models — face/hand/person detection model benchmarks

Related: AI Image Generation Workflows Related: VRAM Optimization for Stable Diffusion Related: Flux Model Guide Related: ADetailer Face Enhancement Related: Best Upscaler Models Compared