TL;DR

• Add PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:512 to webui-user.bat to eliminate most CUDA OOM errors.
• Train FLUX LoRAs on 8–12 GB GPUs with --blocks_to_swap 35 --fp8_base and T5XXL fp8 checkpoints; expect ~4 GB at 512², ~6 GB at 768², ~7.5 GB at 1024².
• Run Wan2.1-T2V-1.3B (8.19 GB) and HunyuanVideo FP8 (13 B params) on 12 GB cards; enable ComfyUI --enable-dynamic-vram for automatic offloading.

PYTORCH_CUDA_ALLOC_CONF and VRAM Optimization for FLUX, Wan2.1, and HunyuanVideo

The PYTORCH_CUDA_ALLOC_CONF environment variable controls PyTorch's CUDA memory allocator through garbage_collection_threshold and max_split_size_mb. Community guides for Stable Diffusion WebUI recommend garbage_collection_threshold:0.9,max_split_size_mb:512, but this variable does not appear in the official PyTorch CUDA semantics documentation (source). Treat this as a community-validated workaround, not a supported API.

Modern diffusion and video models expose model-specific levers: block swapping for FLUX, FP8 quantization for Wan2.1 and HunyuanVideo, and dynamic VRAM modes in ComfyUI. This article consolidates verified numbers, flags, and mechanisms so you can match a model to your GPU without trial-and-error OOM crashes.

Frequent Questions

Model-Specific VRAM Requirements

Note: The FLUX numbers come from a single community benchmark on an RTX 3060 12 GB (Civitai #9487). They are widely cited but not independently reproduced. Keep ≥0.5 GB headroom above the listed values.

Technical Mechanisms

Why blocks_to_swap Reduces VRAM in FLUX Training

--blocks_to_swap offloads transformer blocks from GPU VRAM to system RAM. Each swapped block saves VRAM proportional to its activation size but adds CPU↔GPU transfer latency. FLUX.1 exposes up to 35 swappable blocks; Kohya docs state: "Higher numbers save more VRAM but reduce training speed" (flux_train_network.md).

Why Wan-VAE Cuts Video VRAM

Wan-VAE uses a 3D causal convolution (CausalConv3D) to compress pixel-space video into a latent grid with ratios 4× (temporal) / 8× (spatial) / 16× (channel). This enables unlimited-length 1080p generation without scaling VRAM linearly with clip duration (Wan2.1 model card).

Why FP8 Halves HunyuanVideo Weight Memory

FP8 (E4M3FN) stores each parameter in 8 bits instead of 16. HunyuanVideo's 13 B+ parameters drop from ~26 GB to ~13 GB weight memory. The repo ships run_sample_video_fp8.sh and fp8_optimization.py for one-command FP8 inference (Tencent/HunyuanVideo).

Why PYTORCH_CUDA_ALLOC_CONF Affects OOM

PyTorch's CUDA allocator holds freed blocks in a cache. garbage_collection_threshold sets the fraction of reserved memory that triggers a cache sweep; max_split_size_mb caps the size of a single allocation before the allocator splits it. The community-prescribed 0.9 / 512 keeps the cache lean and prevents large monolithic allocations that fragment VRAM. Not documented in official PyTorch CUDA notes (source).

Common Errors and Fixes

ComfyUI Dynamic VRAM Flags

Source: comfy/cli_args.py in ComfyUI repo (GitHub).

FLUX Training: Practical Flag Combinations

Download T5XXL fp8: t5xxl_fp8_e4m3fn.safetensors from comfyanonymous/flux_text_encoders (Hugging Face).

Gaps and Unverified Claims

1. PYTORCH_CUDA_ALLOC_CONF official docs — Not found on PyTorch CUDA notes page; may exist only in source or Docker docs.
2. FLUX 768² / 1024² VRAM — Single-source community benchmark; no independent replication found.
3. VRAM Estimator plugin accuracy — Not independently validated.
4. HunyuanVideo single-GPU VRAM — README covers multi-GPU and FP8 but not a single-GPU baseline.
5. FLUX.1-dev exact file size — Model card confirms gated safetensors; byte count not sourced.
6. SageAttention benchmark details — README claims 2–5× speedup with accuracy parity; full tables not read.

FAQ

How much VRAM does FLUX.1-dev need for inference? Inference VRAM is lower than training. Community reports ~10–12 GB for fp16/bf16 at 1024² with ComfyUI dynamic VRAM off; GGUF quantizations from city96/FLUX.1-dev-gguf drop this to 6–8 GB (Hugging Face).

What is PYTORCH_CUDA_ALLOC_CONF and how do I set it? An environment variable read by PyTorch's CUDA allocator at process start. On Windows: set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:512 in webui-user.bat. On Linux/macOS: export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:512 in your shell rc or launch script.

Does ComfyUI support dynamic VRAM allocation? Yes. Launch with --enable-dynamic-vram to allow automatic model offloading when VRAM is tight. Complementary flags (--highvram, --gpu_only, --novram, --cpu) select static strategies (ComfyUI cli_args.py).

Can Wan2.1 run on an RTX 3060 12 GB? Yes. The T2V-1.3B model requires 8.19 GB VRAM per its model card, leaving headroom on a 12 GB card (Wan2.1 model card).

What does blocks_to_swap do in FLUX LoRA training? It moves up to 35 transformer blocks between CPU RAM and GPU VRAM per iteration. More blocks swapped = lower peak VRAM, longer step time. Start at 35 for 8 GB cards, reduce as VRAM allows (Kohya flux_train_network.md).

Where do I get T5XXL fp8 checkpoints for low-VRAM FLUX training? comfyanonymous/flux_text_encoders hosts t5xxl_fp8_e4m3fn.safetensors. Load via ComfyUI's DualClipLoader node (Hugging Face).

How does SageAttention compare to FlashAttention? SageAttention's quantized attention kernel claims 2–5× speedup over FlashAttention with no end-to-end metric loss across language, image, and video models (SageAttention GitHub). Integration requires a custom node or patched attention backend.

Key Takeaways

1. Allocator config works but isn't documented — PYTORCH_CUDA_ALLOC_CONF is a community-standard fix for OOM with no official PyTorch reference.
2. FLUX training fits on 8–12 GB — Combine --blocks_to_swap, --fp8_base, and T5XXL fp8 to hit ~4 GB at 512².
3. Video models are VRAM-efficient by design — Wan-VAE's 4/8/16 compression and HunyuanVideo's FP8 weights target consumer GPUs.
4. ComfyUI dynamic VRAM is a first-class feature — Toggle it via CLI; no code changes needed.
5. Community benchmarks fill documentation gaps — Treat single-source numbers as starting points, not guarantees.

Sources

• PyTorch CUDA semantics notes — official CUDA documentation; does not document PYTORCH_CUDA_ALLOC_CONF
• ComfyUI cli_args.py — ComfyUI dynamic VRAM flag definitions
• Kohya sd-scripts flux_train_network.md — FLUX training flags: --blocks_to_swap, --fp8_base, T5XXL fp8
• Wan2.1-T2V-1.3B model card — 8.19 GB VRAM requirement, Wan-VAE compression ratios
• Tencent/HunyuanVideo — FP8 weights release, 13 B+ parameters, multi-GPU sequence parallel
• SageAttention — 2–5× speedup claim over FlashAttention
• Civitai #194 — PYTORCH_CUDA_ALLOC_CONF config tip, VRAM unload button
• Civitai #9487 — FLUX training VRAM measurements at 512²/768²/1024²
• FLUX.1-dev-gguf — GGUF quantized FLUX inference
• flux_text_encoders — T5XXL fp8 checkpoints

• PyTorch CUDA memory management deep dive
• FLUX LoRA training from scratch
• ComfyUI performance tuning checklist
• Quantization guide for diffusion models
• Multi-GPU inference with HunyuanVideo