Stable Diffusion from Scratch in PyTorch | Unconditional Latent Diffusion Models

In this video, we'll cover everything from the building blocks of stable diffusion to its implementation in PyTorch and see how to build and train Stable Diffusion from scratch.

This is Part I of the tutorial where I explain latent diffusion models specifically unconditional latent diffusion models. We dive deep into what is latent diffusion , how latent diffusion works , what are the components and losses used in training latent diffusion models and then finally implement and train latent diffusion models.

The second part will cover conditional latent diffusion models and we will transition to Stable diffusion. The series will be a stable diffusion guide from scratch and you will be able to code stable diffusion in pytorch by yourself by end of it.

⏱️ Timestamps

00:00 Intro
00:56 Recap of Diffusion Models
3:31 Why Latent Diffusion Models
4:31 Introduction to Latent Diffusion Models
5:55 Review of VAE : Variational Auto Encoder
6:29 Review of VQVAE : Vector Quantised Variational Auto Encoder
7:36 Issue with L2 Reconstruction Loss for Latent Diffusion Models
8:50 Perceptual Loss
13:44 LPIPS Implementation
16:40 Adversarial Loss in Latent Diffusion Models
19:38 AutoEncoder Architecture in Latent Diffusion Models
23:04 VAE Implementation vs VQVAE Implementation
24:22 Autoencoder Implementation for Latent Diffusion Models
32:23 Training AutoEncoder for Latent Diffusion Models
32:37 Discriminator for Latent Diffusion Models
36:33 Results of Autoencoder Training
37:40 VQGAN = VQVAE + LPIPS + Discriminator
38:26 Latent Diffusion Model Architecture
39:37 Training Diffusion of Latent Diffusion Models
41:11 Latent Diffusion Model Results
41:45 Whats Next
42:18 Outro

🔔 Subscribe :

📌 Keywords:
#stablediffusion #stable_diffusion

[vikramsandu]

Don't have enough words to describe it. This is presented and explained so beautifully. Thanks, Legend.

[winterknight]

Hands down the best underrated channel right now. I hope you get millions of subscribers soon. The amount of effort and the level of detail all in one video is amazing! Thank you!

[ActualCode]

Nice one, can't wait for the conditional diffusion model.

[himanshurai]

Great work once again! Thanks for the great explanation :)

[bhai_ki_clips]

Amazing explanation and Implementation. Thanks you so much.

[AniketKumar-dlou]

Thank you for such a wonderful explaination.😃

[computing_T]

denoising papers channel I could say! Thanks a lot TK🙏🙏

[_divya_shakti]

Hi Tushar . . . Amazing lecture thankyou . . . ❤❤❤ Lot of love for this video . . .. looking forward for more advance content in text to image generation . . . . I am researching in making some combo of GANs and existing SD model weights such that we can generate some low quality image or latent on low end devices like CPU and then upscale low res image using some GANs . . . Trying to implement something kind of mobile diffusion . . . . Please suggest some good resource to move forward 😅

[chickenp]

why do we do retain_graph=True in the VAE training script?

[rafa_br]

Awesome, I just have one question, why not only use the adversarial loss? (or maybe first train the VAE using normal L1/L2 and then fine tune it using adversarial loss) I feel like using all at once would just end up consuming more resources.
Or would it cause the VAE to pretty much inherit all problems GANs have (mode collapse, vanishing gradients, etc)?

[Swatisd]

Thanks for the video, it is really great. I have finetuned a stable diffusion v1.5 model and now I am trying to built a stable diffusion model from scratch without using any pretrained ckpts and running it locally. So is it possible that we can train the model without using any pretrained checkpoint ?

[HardikBishnoi]

What was the compute you used to train this? And how long did it take? Great video btw!

[jaszczurix]

Does this video explain the training of the entire model in which I can enter the prompts to generate images?

[nadavpotasman]

thanks a lot! can you explain why you didn't use SA in the AE implementation? you mentioned that it's related to the images' high resolution, why? why you do use SA in he DDMP implementation?

[paktv]

thanks for explanation. i have a question you said the autoencoder takes the image in pixel space generate the latent space and decoder do the reverse. if we train the diffusion model it will take the noise sample of the latent space and then goes through the unet which work as a noise predictor and remove the noise from the noisy latenct space iamges from the encoder side.
my question is how and from where it will add nosie to the latent space and how the Unet will do the diffusion process to remove the noise because you also said that we don't need the Time step information?
kindly i have these doubts if you can address it kindly clear my doubts. Thanks for the making video on this topic!

[MilesBellas]

Try a version with an AI voice for clarity ?