Step-by-Step Stable Diffusion with Python [LCM, SDXL Turbo, StreamDiffusion, ControlNet, Real-Time]

In this step-by-step tutorial for absolute beginners, I will show you how to install everything you need from scratch: create Python environments in MacOS, Windows and Linux, generate real-time stable diffusion images from a webcam, use 3 different methods including Latent Consistency Models (LCMs) with ControlNet, SDXL Turbo, and StreamDiffusion. We will use controlnet, image2image, or essentially image2video in 1 or 2 inference steps (essentially real-time!) We will learn how to create a hack to generate faster with Macbooks that have MPS, computers with CUDA GPUs, and even lower-end computers that only have CPU power! Get ready to generate AI for free without MidJourney without DALL-E and without any third-party website! While Automatic1111 and ComfyUI are amazing programs, in this tutorial, we will learn how to do everything with code – so if you like to hack, then this is FOR YOU – but, even if you are used to Automatic1111 and ComfyUI, this will be a great tutorial to learn how to do things through code! Go crazy with latent diffusion and run with what you learn and build your own projects. All code is FREE and downloadable!

**CODE FROM TUTORIAL**

** WINDOWS CUDA SPECIFIC TUTORIAL (5 MIN TUTORIAL)

** CHAPTERS **

00:00 - Demo and Overview
00:44 - Automatic1111 and ComfyUI vs Python Code
01:07 - Overview of LCM, SDXL Turbo and StreamDiffusion
01:46 - Get Inspired, Proyector Demo, and Ideas
02:23 - Quick Visual Explanation of Diffusion Models
03:24 - How To Download: GIT LFS, VS Code (or text editor), Python
04:44 - Creating Python Environment for LCM and SDXL Turbo
05:22 - Creating Python Environment for StreamDiffusion
07:28 - Stable Diffusion Models Installation
09:59 - Webcam code overview
14:55 - LCM, Controlnet and SDXL Turbo code overview
24:20 - Running the LCM Demo
26:41 - Running SDXL Turbo Demo
28:26 - Activating StreamDiffusion venv
29:11 - StreamDiffusion code overview
29:29 - StreamDiffusion with CUDA
30:04 - StreamDiffusion MPS hack
31:45 - StreamDiffusion script prep
32:27 - Running StreamDiffusion Demo 1
33:17 - Running StreamDiffusion Demo 2
34:20 - Running StreamDiffusion Demo 2 adding props
35:12 - Running StreamDiffusion Demo 3
35:31 - Running StreamDiffusion Demo 3 adding props
36:06 - BONUS: How To Do More Cool AI

**DOWNLOAD LINKS**

Install Python 3.11.6 for the LCM and SDXL Turbo
Install Python 3.10.11 for streamdiffusion

For Windows, to check which Python versions you have in command prompt:

"py -0"

** CREATING ENVIRONMENTS IN WINDOWS **
[Consider changing the directory you install the environments to one that is not in C. Change to different drive with something like “d:” and then “cd (your username)\Documents]

For LCM and SDXL Turbo:
"py -3.11 -m venv lcm_sdxlturbo_venv"

For StreamDiffusion:
"py -3.10 -m venv streamdiffusion_venv"

To activate:

For LCM and SDXL Turbo:

For StreamDiffusion:

***QUICK COMMANDS***

Ctrl + c on Mac to quit the program running
Ctrl + c on Windows to quit program (if this doesn’t work, try ctrl + pause/break)

“ls” to list files on Mac and “dir” to list files on Windows

***MODELS***

***BOOK ON HOW TO USE AI FOR HACKERS AND MAKERS***

PRE-ORDER DIY AI BOOK by my brothers and me!:

The book goes into deeper explanations of how latent diffusion works, how to write modular code and how to get it to work smoothly on any system. So pre-order to have a ton of detailed and explained resources for anything AI!

Camera to AI with python
Realtime image2video
Realtime image to video
Video to video
Single step inference
1 step inference stable diffusion
stable diffusion image2image
realtime stable diffusion
Stable diffusion sdxl turbo
Stable diffusion LCM
Stable diffusion streamdiffusion
Super fast generative ai
Huggingface diffusers
Stable diffusion controlnet
How to animate with diffusers
Animatediff control net
Animatediff animation
Stable Diffusion animation
comfyui animation
animatediff stable diffusion
Controlnet animation
how to use animatediff
animation with animate diff comfyui
how to animate in comfy ui
prompt stable diffusion
animatediff comfyui video to video
animatediff comfyui install
animatediff comfyui img2img
animatediff vid2vid comfyui
DreamShaper
Epic Realism
comfyui-animatediff-evolved
animatediff controlnet animation in comfyui

[njkross]

This is a near perfect example of what tutorial videos should be like

[Rolungo]

this was awesome and super easy to follow. thanks for making this accessible.

[jonbonacorsi]

Very great tutorial, i bought the scripts. Perfect !

[oceanradiostation]

Music is too loud in this video, but the content is very valuable!

[sdsld]

This was very easy to understand and follow and a fun project too. My only question is how to enable the safety checker if you have this in like a public place? I found lots of resources on how to disable it, but not to enable, lol.

Also I like your diy AI book, but if I may suggest to rather than publish a paper book, maybe consider making a class on skillshare or something like that? This way users will have access to only resources and links (like in this video), also I looked at some of the content covered and noticed the release date of the book (near the end of summer), most of tools and content will most certainly be replaced with improved models or workflows by that time. I would just imagine updating an online skillshare class or adding another course section would be easier to maintain up-to-date material (since most of your customers will likely be AI enthusiasts they'll be on the computer anyway). Just a suggestion! I would be interested in something like this.

Also I appreciate your inclusion of Mac OS users in your video, of course cuda will always be faster but for experimental and learning purposes I'm happy to see this. I hope you keep a bilateral approach when developing new content so people from across platforms can learn this tech with equity.

[AI_Guy-zqxs]

Thank you for this great tutorial, I have followed this and the short windows version. Just wondering if you have any tips on how I can enlarge the mask / generated image overlay, for example 1024x1024, anything I have tried results in an error along the lines of "ValueError: could not broadcast input array from shape (1024, 1024, 3) into shape (152, 1024, 3)". I'm not overly bothered if the generated image is 512 x 512 and stretched I just want the area to cover more of the webcam feed.

[racrosoft]

where can I get the two text files from?

[pablqwh]

where do I get the twos .txt files from??

[jonmichaelgalindo]

Can't hear you over the people shouting and metal crashing.

[navyyard]

result_image[center_y:center_y+HEIGHT, center_x:center_x+WIDTH] = cv2.cvtColor(np.array(rendered_image), cv2.COLOR_RGB2BGR)

ValueError: could not broadcast input array from shape (512, 512, 3) into shape (136, 96, 3)

I have been getting this error on windows. No Code change. for stream diffusion

[navyyard]

result_image[center_y:center_y+HEIGHT, center_x:center_x+WIDTH] = cv2.cvtColor(np.array(rendered_image), cv2.COLOR_RGB2BGR)

ValueError: could not broadcast input array from shape (512, 512, 3) into shape (136, 96, 3)

I have been getting this error on windows. No Code change for stream diffusion