Transform Your Sketches into Masterpieces with Stable Diffusion ControlNet AI - How To Use Tutorial

Playlist of #StableDiffusion Tutorials, Automatic1111 and Google Colab Guides, DreamBooth, Textual Inversion / Embedding, LoRA, AI Upscaling, Pix2Pix, Img2Img:

Paper Adding #Conditional Control to Text-to-Image Diffusion Models :

0:00 What is revolutionary new Stable Diffusion AI technology ControlNet
0:36 What is ControlNet with Canny Edge
0:49 What is ControlNet with M-LSD Lines
1:17 What is ControlNet with HED Boundary
1:41 What is ControlNet with User Scribbles
1:58 What is ControlNet Interactive Interface
2:08 What is ControlNet with Fake Scribbles
2:28 What is ControlNet with Human Pose
2:45 What is ControlNet with Semantic Segmentation
3:02 What is ControlNet with Depth
3:15 What is ControlNet with Normal Map
3:35 How to download and install Anaconda
4:33 How to download / git clone ControlNet
5:25 How to download ControlNet models from Hugging Face repo
6:37 Which folder is the correct folder to put ControlNet models
7:03 How to install ControlNet to generate virtual environment with correct dependencies
8:53 How to start run first app Canny Edge
9:59 Correct local URL of the app
10:11 Testing first test image bird with Canny Edge
11:42 How to start M-LSD lines ControlNet app
12:10 How to set low VRAM option in configuration
13:20 Start again M-LSD lines ControlNet app
13:37 Running Hough Line Maps app example
14:28 Example of Control Stable Diffusion with HED Maps
14:45 Testing ControlNet with User Scribbles

Used lineart source :

From official paper of Adding Conditional Control to Text-to-Image Diffusion Models
We present a neural network structure, ControlNet, to control pretrained large
diffusion models to support additional input conditions. The ControlNet learns
task-specific conditions in an end-to-end way, and the learning is robust even when
the training dataset is small ( 50k). Moreover, training a ControlNet is as fast as
fine-tuning a diffusion model, and the model can be trained on a personal devices.
Alternatively, if powerful computation clusters are available, the model can scale to
large amounts (millions to billions) of data. We report that large diffusion models
like Stable Diffusion can be augmented with ControlNets to enable conditional
inputs like edge maps, segmentation maps, keypoints, etc. This may enrich the
methods to control large diffusion models and further facilitate related applications.

1 Introduction
With the presence of large text-to-image models, generating a visually appealing image may require only a short descriptive prompt entered by users. After typing some texts and getting the images, we may naturally come up with several questions: does this prompt-based control satisfy our needs? For example in image processing, considering many long-standing tasks with clear problem formulations, can these large models be applied to facilitate these specific tasks? What kind of framework should we build to handle the wide range of problem conditions and user controls? In specific tasks, can large models preserve the advantages and capabilities obtained from billions of images? To answer these questions, we investigate various image processing applications and have three findings. First, the available data scale in a task-specific domain is not always as large as that in the general image-text domain. The largest dataset size of many specific problems (e.g., object shape/normal, pose understanding, etc.) is often under 100k, i.e., 5 × 104 times smaller than LAION5B. This would require robust neural network training method to avoid overfitting and to preserve generalization ability when the large models are trained for specific problems. Second, when image processing tasks are handled with data-driven solutions, large computation clusters are not always available. This makes fast training methods important for optimizing large models to specific tasks within an acceptable amount of time and memory space (e.g., on personal devices).

[SECourses]

Please join discord, mention me and ask me any questions. Thank you for like, subscribe, share and Patreon support. I am open to private consulting with Patreon subscription.

[cedricsyllebranque]

These are awesome! So many applications: better, style transfer, coloring, ... Thanks for help! Waiting for Automatic1111 version...

[bayz]

Thanks for the great tutorial! However, I'm having problem using the ControlNet after successfully installed. When I run it, it shows: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select). Has anyone seen the same error?

[ftamash]

This is amazing, thank you!
I just can't get the code repository updated by the git pull repo_URL command, i got this error :
"fatal: 'repo_URL' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists."

Any idea how to fix this?
Thank you !!

[kanall]

I have a "ControlNet" folder with "model" folder inside with pixar, monaliza, etc in pt. files how use it ?please

[TV---knrl]

Hello! I keep getting the error: "FileNotFound [Errno2] No such file or directory" when trying to input each of the models. Solution? Thanks!

[vanessadpr]

incredible ! thank you so much for this step by step tutorial ! I'm wondering how to run it using my second graphic cards (i have a SLI) because now it uses only the first GTX 1080, but my second is idle. But maybe GTX 1080 is too old now for that ? I tried to install tensorflow-gpu on anaconda but it fail installing it

[monali]

It works fine! Thank you so much for showing step-by-step how to install it properly. Downloading the ControlNet models takes a long time, but it's definitely worth the trouble. I have just one question: the created images are not automatically saved, are they?

[DofxStudio]

Hocam canny modelini yükledikten sonra direkt 127.0.0.1'e geçtin. öncesinde bir bat dosyası çalıştırmamız gerekmiyor mu? örneğin scribble2image çalışmıyor. no module xformer ve RuntimeError Cuda hataları geliyor.

[sadelcri]

someone already made an extension to implement it to automatic1111. Could you make a tutorial on its usage?

[OrosAIArt]

It would be great if it was available as a plugin for automatic 1111

[shortandcut]

I was a virgin when I saw the stable diffusion group on reddit, and it so hard to learn, because of you now I am deflowered and have experience. Thank you for your guides!

[HelyRojas]

HI
I followed all the steps of the tutorial.
But when executing the command: python gradio_canny2image.py
I get this error: (My PC Intel(R) Core(TM) i7-6700 CPU. NO GPU, )

raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with to map your storages to the CPU.

[thedisciplinedweeb]

for youtube algorythm love your content

[rafis]

and you can vice versa a picture in a sketch drawing?

[diego.spirit]

when I put a black and white drawing, it generates another black and white drawing, not colored, how do I solve it?

[karlderkafer]

If you don't have a lot of VRAM (less than 12 GB) you better use the ControlNet Extension for Automatic1111 WebUI (next video in playlist)

[nancybuckle]

Help w/ setting up on Colab would be greatly appreciated. Ur vids are great btw.

[snoweh]

Jesus 88GB of models. I wonder if you were having trouble with 3060 12GB, would I have a problem with 3080 10GB.

[Sander_He]

Hi, I am a macOS user. When I run conda env create -f environment.yaml, I get an error ResolvePackageNotFound: - cudatoolkit=11.3. How can I solve this problem?🥲