AnimateDiff Video-to-Video: Flicker & ControlNet

Turning an existing clip into a stylised animation with AnimateDiff comes down to three technical decisions: how you prepare the source clip, how you suppress frame-to-frame flicker, and which ControlNet stack keeps the motion legible. This guide walks through each layer with settings verified against the official AnimateDiff repository [https://github.com/guoyww/AnimateDiff], ByteDance's Lightning model card [https://huggingface.co/ByteDance/AnimateDiff-Lightning], ControlNet documentation [https://github.com/lllyasviel/ControlNet], and the community workflows published on Civitai. For related background, see our ControlNet Complete Guide and Low-VRAM Stable Diffusion Guide.

TL;DR

• Prep the clip: crop to ≤512×768 (or 576×1024 for Lightning) [https://huggingface.co/ByteDance/AnimateDiff-Lightning], loop it cleanly with LoopMaker [https://github.com/markuryy/LoopMaker], and keep it under 8 seconds.
• Kill flicker: set noise multiplier to 0 in Mov2Mov [https://civitai.com/articles/1961], pair lineart_realistic with control_v11p_sd15_lineart at weight 0.6, and run DPM++ 2M Karras at 20 steps, CFG 11, denoise 0.6.
• Stack ControlNet: layer lineart for structure, OpenPose or Depth for pose, and optionally IP-Adapter for face consistency; the AnimateDiff ControlNet checkpoint [https://huggingface.co/crishhh/animatediff_controlnet] handles the second k-sampler pass in Lightning workflows. Learn more about the underlying architecture in ControlNet Explained.

How Do You Prepare a Source Clip for AnimateDiff Video-to-Video?

Resolution and length are the hard constraints. The low-VRAM community workflow (RTX 3060, 8 GB) recommends 512×768 maximum; generating larger "may lead to errors, no matter how much VRAM you have" [https://civitai.com/articles/3518]. The AnimateDiff-Lightning team tested at 576×1024, 8 seconds, 30 fps and warns "video shouldn't be too long or too high resolution" [https://huggingface.co/ByteDance/AnimateDiff-Lightning]. Keep your clip under 8 seconds and match the aspect ratio of your target resolution.

If the source isn't already a seamless loop, run it through LoopMaker [https://github.com/markuryy/LoopMaker], a Python app and Colab notebook that "automatically identify and create looping segments from video files." For broader context, see Video Preprocessing for Animation.

Which Settings Reduce Flicker Between Frames?

Flicker comes from different noise being injected into each frame's latent. The Mov2Mov workflow states: "We can leave the Noise multiplier to 0 to reduce flickering" [https://civitai.com/articles/1961].

The same workflow pairs zero noise with a lineart ControlNet stack:

The lineart preprocessor extracts edges; the ControlNet model conditions the diffusion to follow those edges at weight 0.6. ControlNet is composable: "more than one ControlNet can be easily composed to multi-condition control" [https://github.com/lllyasviel/ControlNet]. A practical video-to-video stack layers:

1. Lineart (control_v11p_sd15_lineart) — frame edges, weight ~0.6.
2. OpenPose (control_v11p_sd15_openpose) or Depth — body pose and spatial layout. The Lightning v2v workflow ships with OpenPose [https://huggingface.co/ByteDance/AnimateDiff-Lightning].
3. IP-Adapter (optional) — face consistency across frames. Lightning tips mention "IP-Adapter for improved face consistency" [https://huggingface.co/ByteDance/AnimateDiff-Lightning].
4. AnimateDiff ControlNet checkpoint (crishhh/animatediff_controlnet) [https://huggingface.co/crishhh/animatediff_controlnet] — used in the second k-sampler pass of the Lightning ComfyUI workflow for temporal coherence.

ControlNet copies network blocks into a locked copy (preserving the model) and a trainable copy (learning the control signal). Zero convolutions — 1x1 convolutions with weights and bias initialized as zeros — start neutral and ramp up during training [https://github.com/lllyasviel/ControlNet].

Low-VRAM Setup (RTX 3060 / 8 GB)

The community guide for sub-4090 cards uses these webui-user.bat flags [https://civitai.com/articles/3518]:

set COMMANDLINE_ARGS=--xformers --upcast-sampling --lowvram
set CUDA_MODULE_LOADING=LAZY
set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:32

• --xformers reduces memory consumption and increases speed [https://civitai.com/articles/3518].
• --lowvram reduces GPU memory usage but greatly reduces generation speed: the GPU stores only what is needed for each step, transferring data to RAM between steps (22.9 GB system RAM in the guide's 16-frame test) [https://civitai.com/articles/3518].
• --upcast-sampling fixes the "Upcast cross attention layer to float32" error that can appear with xformers [https://civitai.com/articles/3518].

With >6 GB VRAM, switch --lowvram to --medvram after the first run [https://civitai.com/articles/3518].

Generation at 512×768, 16 frames, 8 fps takes roughly one hour on an RTX 3060 [https://civitai.com/articles/3518].

AnimateDiff Lightning: Faster Video-to-Video

ByteDance's AnimateDiff-Lightning distills the original motion module into 1-, 2-, 4-, and 8-step checkpoints [https://huggingface.co/ByteDance/AnimateDiff-Lightning]. The 8-step variant (animatediff_lightning_8step_comfyui.safetensors) delivers the best quality; 2-step and 4-step variants deliver faster inference at lower quality.

Recommended Lightning settings:

The Lightning v2v workflow adds ControlNet OpenPose (control_v11p_sd15_openpose.pth) and the AnimateDiff ControlNet checkpoint [https://huggingface.co/crishhh/animatediff_controlnet] for the second pass.

Extending Length and Changing Content

• Frame interpolation: Enable Deforum's FILM interpolation at Interp X = 10 to slow the video 10× (30 frames at 30 fps → 10 seconds) [https://civitai.com/articles/3518].
• Looping: Use LoopMaker [https://github.com/markuryy/LoopMaker] to create a seamless source clip, then enable AnimateDiff's Closed loop with R+P mode.

Common Errors and Fixes

FAQ

How do I reduce flickering in AnimateDiff video-to-video?

Set the noise multiplier to 0 in Mov2Mov, use the lineart ControlNet at weight 0.6 (Balanced mode), and run DPM++ 2M Karras at 20 steps with CFG 11 and denoise 0.6 [https://civitai.com/articles/1961].

What ControlNet model works best with AnimateDiff for animating video?

control_v11p_sd15_lineart for edges, control_v11p_sd15_openpose for pose, and the AnimateDiff ControlNet checkpoint [https://huggingface.co/crishhh/animatediff_controlnet] for the second k-sampler pass in Lightning workflows. Stack them — ControlNet is composable by design [https://github.com/lllyasviel/ControlNet].

How to set up AnimateDiff on a low-VRAM GPU (e.g., RTX 3060)?

Add --xformers --upcast-sampling --lowvram to webui-user.bat, plus CUDA_MODULE_LOADING=LAZY and PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:32 [https://civitai.com/articles/3518]. Expect about one hour for 16 frames at 512×768. Switch to --medvram with >6 GB VRAM [https://civitai.com/articles/3518].

What is the difference between AnimateDiff motion module v14 and v15?

Community terminology: v14 is described as "more stable," v15 as "larger." The official repo does not version modules this way; the standard file is mm_sd_v15_v2.ckpt [https://civitai.com/articles/3518]. For Lightning, use the dedicated 8-step checkpoint from ByteDance [https://huggingface.co/ByteDance/AnimateDiff-Lightning].

Can I use AnimateDiff Lightning for video-to-video?

Yes. The Lightning repo provides a ComfyUI v2v workflow using ControlNet OpenPose and the AnimateDiff ControlNet checkpoint [https://huggingface.co/crishhh/animatediff_controlnet]. Tested at 576×1024, 8 seconds, 30 fps [https://huggingface.co/ByteDance/AnimateDiff-Lightning]. Use Euler sampler, SGM uniform scheduler, CFG 2, and the 8-step motion module.

How do I make the output video loop seamlessly?

Use LoopMaker [https://github.com/markuryy/LoopMaker] to extract a looping segment from the source, then enable AnimateDiff's "Closed loop" with R+P mode.

What resolution should my source clip be?

512×768 maximum for the standard WebUI workflow [https://civitai.com/articles/3518]; 576×1024 for AnimateDiff-Lightning [https://huggingface.co/ByteDance/AnimateDiff-Lightning]. Larger resolutions cause errors regardless of VRAM [https://civitai.com/articles/3518].

Sources

1. AnimateDiff official repository [https://github.com/guoyww/AnimateDiff]
2. AnimateDiff-Lightning model card [https://huggingface.co/ByteDance/AnimateDiff-Lightning]
3. AnimateDiff ControlNet checkpoint [https://huggingface.co/crishhh/animatediff_controlnet]
4. ControlNet repository [https://github.com/lllyasviel/ControlNet]
5. LoopMaker [https://github.com/markuryy/LoopMaker]
6. ThinkDiffusion Mov2Mov guide [https://civitai.com/articles/1961]
7. Low-VRAM AnimateDiff guide [https://civitai.com/articles/3518]
8. AnimateDiff Lightning dance workflow [https://civitai.com/articles/4769]