Variational Autoencoder from scratch in PyTorch

❤️ Support the channel ❤️

Paid Courses I recommend for learning (affiliate links, no extra cost for you):

✨ Free Resources that are great:

💻 My Deep Learning Setup and Recording Setup:

GitHub Repository:

✅ One-Time Donations:

▶️ You Can Connect with me on:

Useful article on VAEs:

Timestamps:
0:00 - Introduction
2:45 - Model architecture
15:50 - Training loop
31:10 - Inference example
39:10 - Ending

[nikitaandriievskyi]

100% Agree that if you write everyting from scratch line by line it is much better than having it prewritten

[pheel_flex]

Great video, thank you! Please don't change to having pre-written code. Your approach is the best that can be found these days.

[marcocastangia]

Great content. I've always loved your "from scratch" tutorials.

[chrisoman]

Hey just watched your video, really good! But its obvious this is a new area for you (which is not bad), so I thought I'd give you some pointers to improve your algorithm.

1. In practice VAE's are typically trained by estimating the log variance not the std, this is for numerical stability and improves convergence of the results so your loss would go from:
`- torch.sum(1 + torch.log(sigma.pow(2)) - mu.pow(2) - sigma.pow(2))` ->
`-0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp()`
(where log_var is the output of your encoder, also your missing a factor 0.5 for the numerically stable ELBO)
Also, the ELBO is the Expectation of the reconstruction loss (the mean in this case) and the negative sum of the KL divergence

2. The ELBO (the loss) is based on a variational lower bound its not just a 2 losses stuck together as such arbitrarily weighting the reconstruction loss and the KL divergence will give you unstable results, that being said your intuition was on the right path. VAEs are getting long in the tooth now and there are heavily improve versions that focus specifically on "explainable" if you want to understand them I would look at the Beta-VAE paper (which weights the KL divergence) then look into Disentagled VAE (see: "Structured Disentangled Representations", "Disentangling by Factorising") these methodologies force each "factor" into a normal Gaussian distribution rather than mixing the latent variables. The result would be for the MNIST with a z dim of 10 each factor representing theoretically a variation of each number so sampling from each factor will give you "explainable" generations.

3. Finally your reconstruction loss should be coupled with your epsilon (your variational prior), typically (with some huge simplifications) MSE => epsilon ~ Gaussian Distribution, BCE => epislon ~ Bernoulli distribution

[nathantrance]

You are truly a life saver sir. Thank you for keeping everything simple instead of using programming shenanigans just to make it more complicated and unreadable.
Love your tutorials, I learned a lot from your line of thinking, including the ranting things.

[tode]

Again an awesome from-scratch video! I have never seen programming videos in which it is so simple to follow what the person is coding, thank you.
Currently, there are no videos about stable diffusion from scratch, which include the training scripts.
It would be great to see a video on this!

[TsiHang]

I won't have passed my fyp and graduated without you. THANK YOU

[Uminaty]

As usual, it's amazing content ! Thank you so much for your work

[mcemgunes]

Just Great! From-scratch videos are the best to understand the overall flow! Thanks!

[donfeto]

23:09 since you used sigmoid your pixels will be between 0 and 1 so it's okay to use sigmoid in this case otherwise if you use no activation function in the last layer of the decoder you need to use the new loss function of MSE +Reconstartion loss
that what i think

[oceanwave]

10:03 I think the sigma here, which is the output of "self.hid_2sigma", should be interpreted as "log of variance". Why? Because the Linear layer can somehow output negative values while Variance (of a distribution) can't be negative. By interpreting as "log of variance", we can get the variance inside by using exp().

As a result, we have two needed code changes: doing the exp() against this "log of variance" when calculating z as well as calculating the Loss.

[starlite]

Awesome work. Please do more stuff with GANs or visual transformers.

[dr_rahmani_m]

I like the thought process. So, thanks for the 'from scratch' tutorials.

[zetadoop]

Your videos are shockingly good! Among programming channels it is the best one.

[kl_moon]

I love "from scratch" series, plz make more videos..!! and thank you so much!!!

[davidlourenco]

I prefer from scratch too for all the reasons you've mentioned. Thanks for the content .

[TsiHang]

Had to learn about VAE with zero experience in coding or ML. Thank God I found this video 😅

[GoldenMunkee]

I just have to say that, even as someone with a Master's in Data Science from a top university, I still use your tutorials for my work and my projects. Your stuff is incredibly helpful from a practical perspective. In school, they teach you theory with little to no instruction on how to actually build anything. Thank you so much for your hard work!!

[СемёнСемёныч-ед]

Code from 15:05 so you don't need to type it all:
import torch
import torchvision.datasets as datasets
from tqdm import tqdm
from torch import nn, optim
from model import VariationalAutoEncoder
from torchvision import transforms
from torchvision.utils import save_image
from torch.utils.data import DataLoader

[CrypticPulsar]

You are incredible.. thanks for sharing all this knowledge and skill with the world..