ComfyUI AnimateDiff Prompt Travel: ControlNets and Video to Video!!!

This is a fast introduction into @Inner-Reflections-AI workflow regarding AnimateDiff powered video to video with the use of ControlNet.

You can download the ControlNet models here:

The workflow file can be downloaded from here:

The model (checkpoint) downloaded for this tutorial series are here:

The VAE used can be downloaded from:

The motion_modules and motion_loras can be found on the original AnimateDiff repo where you will be offered different sources to download them from:

Or here's a quick link to civitai:

Socials:

[yoyok]

Tested on AMD RX6800XT (Ubuntu22.04 + ROCm 5.7). it works flawlessly and stay close to 12Go of VRAM. Really helpful, thanks a lot.

[wholeness]

Bro we on this journey together. Keep goin!

[EM-kj]

Hard! Thank you very much for paying for it all the way! It's a pity that I brushed it here before going to bed and have to wait until tomorrow to practice.

[JaredVBrown]

Very helpful and approachable tutorial. Thanks!

[SkyOrtizCreative]

Love your vids bro!!! I know it takes a lot of work to makes these, really appreciate your efforts. 🙌

Do u know what happen to this error?

Error occurred when executing



File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)

File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)

File "D:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))

File "D:\AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\nodes_extras.py", line 52, in load_checkpoint
out = load_checkpoint_guess_config(ckpt_path, output_vae=True, output_clip=True,

File "D:\AI\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 511, in load_checkpoint_guess_config
model_config = model_detection.model_config_from_unet(sd, diffusion_model_prefix)

File "D:\AI\ComfyUI_windows_portable\ComfyUI\comfy\model_detection.py", line 239, in model_config_from_unet
unet_config = detect_unet_config(state_dict, unet_key_prefix)

File "D:\AI\ComfyUI_windows_portable\ComfyUI\comfy\model_detection.py", line 120, in detect_unet_config
model_channels =

[edkenndy]

Awesome! Thanks for sharing the resources.

[Copperpot]

Nice job on these of late. In general I have a hard time watching video tutorials w/ people on screen talking - but you're hitting all the right notes on these so -wanted- to bother w/ Comfy, but have definitely admired the generations some have been sharing. Thanks for making well timed / friendly tutorials. Stick w/ it and you'll def build a good/active channel. Thanks!

[leretah]

awesome, thank you. Im really appreciated

[Jovis.x]

error
KSampler
mat1 and mat2 shapes cannot be multiplied (2464x2048 and 768x320)

[RenoRivsan]

can you show how to remove animatediff from this workflow... i dont want my video to change style

[lfar]

I use run diffusion and can't load your workflow in json( can you upload the pic too maybe that can fix it?

I don't know how to fix this problem.


'ControlNet' object has no attribute 'latent_format'

[ronnykhalil]

yea baby (edit: this is straight up the most valuable 10 minutes ive watched on YouTube in a while, exactly the signal I needed amidst all the noise regarding comfy and diff. You explained it really well and clear. Thank ye kindly!

[UON]

Exciting! I hope this helps me figure out how to do a much longer vid2vid without running out of vram

[Andro-Meta]

Converting the pretext, and how to do that completely blew my mind and opened doors to understanding what I could do. Thank you.

[victorvaltchev]

What was the size of the video in the end? Because you showed 1024 576 in the beginning. Is that the resolution in the end as well? Also how do you load other formats of video, I only have webp and gif?

[calvinherbst]

Thank you. Excellent tutorial :) Keep them coming, subbed!

[BrandonFoy]

Whoa! This is awesome, thanks for sharing your workflow.

I haven’t used ComfyUI - just been in A1111. Can you recommend tutorials for Comfy? Or any you’ve made that’ll be a solid start to start learning this method?

Thank you!!

[Spajra-music]

followed this all the way through and at the end my video output was just black. any suggestions?