How to Add ControlNet & AnimateDiff Together in ComfyUI

#stablediffusion #aiart #generativeart #aitools #comfyui
ControlNet allows you to add conditions to your prompt to allow for an accurate representation of a subject when being passed into the diffusion model. Here's how to use ControlNet along with AnimateDiff in ComfyUI.

# Time Stamps
Intro: 0:00
Reviewing AnimateDiff: 0:20
Setting Up ControlNet Nodes: 1:03
Passing to Canny Edge: 1:28
Passing to DWPose: 3:33
Reviewing the Output: 6:48
Outro: 8:33

txt2vid in ComfyUI video:

Workflow:

Install ComfyUI Manager (You'll need this in order to install the custom nodes):

ComfyUI-Advanced-ControlNet:

ComfyUI's ControlNet Auxiliary Preprocessors:

ControlNet Models:

Important links:

[ParadoxicLunatic]

Great video, clear, no annoying accent, concise example of how to and what it does. Cleared up some roadblocks I encountered. Gonna check out the rest of your content the coming days and weeks!

[TheGladScientist]

Appreciate the conciseness here. Keep up the good work!

[masoud.art.videos]

Very clear guide thanks. Any advice on how to optimize your GPU to run things like this. I can see you are using Mac. I have a strong windows machine with 64gb GPU but getting very very long processing time for anything more than 10 second output

[GES]

How do you add face swap into this, to control the identity of the character?

[bratbart]

Hi, thx so much for an informative tutorial. I have a problem though. When I use your "vid2vid Using AnimateDiff + ControlNet" workflow I can't get the final video. Everything works including the Ksampler but the last preview window and the video combine node show only black frames. Could you give me advice on this issue? Also do you know how can I include a LORA of a person to this workflow?

[wonderhighlights]

Your app is so excellent, everything in which i am interested is present. I am a LOGO Designer can i make it here? My english is not so good but i hope you understand☺

[datrighttv]

Appreciate. How to make animation but the character has no change clothes, hair, background...