Batch normalization | What it is and how to implement it

In this video, we will learn about Batch Normalization. Batch Normalization is a secret weapon that has the power to solve many problems at once. It is a great tool to deal with the unstable gradients problem, helps deal with overfitting and might even make your models train faster.

Get your free speech-to-text API token 👇

We will first go into what batch normalization is and how it works. Later we will talk about why you might want to use it in your projects and some benefits of it. And lastly, we will learn how to apply Batch Learning to your models using Python and Keras. Even though it is fairly simple to apply Batch Normalization using Keras, we will touch upon some details that might need extra care.

[ludwigstumpp]

Thanks for doing these videos. :)

As someone who is familiar with Batch Normalization I was personally missing a few important information which is why I add them here for the community:

- The normalisation is happening over the batch dimension (in contrast to other variants such as layer normalisation where we normalise over the layer dimension), meaning that we normalize the feature over the mini-batch
- which is why it does not work well for smaller batch sizes (usually 16+)
- another advantage for the scale and the offset parameter is that it allows the network to undo the BN, meaning that BN can't make your result worse
- during test time with e.g one sample only we can't compute mean and std since we don't have a batch. This is why we use running statistics of mean and variance calculated during training

[AlexKashie]

Best explanation of Normalization and Standardization... Thank you

[pauldonzier]

Standardization does not change the overall shape of your distribution, it just translates and scales it to have mean 0 and std 1. It will give a normal distribution if and only if your distribution was already normal. If your distribution was uniform, or poisson, or whatever, it will remain like that

[donmiguel]

The hint to omit the bias when using afterwards batchnorm_layers is very good, the information that batch_norm can be used while omitting the learnable scaling and offset would also be helpfull because these functionality is also computational very expensive and not the core feature of the batch_norm.

[jmf]

I got impression that this video is about normalization. There is nothing about batches and what does it mean BATCH normalization

[VritanshKamal]

Nicely Explained ! I liked the part where we start from definitions.

[spider]

First thanks for an amazing video, answered many questions. It felt like you predicted my questions during the video and proposed the answers right after!

I have one question, if we don't normalize the input and use BatchNormalisation, wouldn't it behave completly different?
Like for example say we feed training images of luminance value of 0-200, but on real world interference or during validation we use some other images that have full scale luminance values 0-255.
Giving we know the range of our luminance values during modeling, wouldn't be better to use prenormalization, as Batch normalization will behave incorrectly during real world/validation process?

P.S. To avoid any confusion about the data and why we didn't feed 0-255 before, say we have grayscale images, and we don't know if they're all in range, and what we'll have during validation, basically random split.

[eyadaiman]

Thank you, Please keep doing this kind of videos, your explanation is simple and clear

[ramiallouch]

simple and very useful . thank you for this great content

[juliusodunuga]

Thank you 👍🏻👍🏻👍🏻
The explanation is superb

[mariajoseapolo]

Thanks for the info, It was super easy to understand and clear. ( :

[nayanparnami]

Superb explanation to one of the important interview question..
Great work.!!👍🏻👍🏻👍🏻
Thanks for the video.

[Omsip]

What I learned from other videos, that all of this is applied across the samples per batch for each weight. Not across all weights. I hope I got it right…

[mayankanand]

I have watched this video over and over, you explained it very well though I am not a fan of Keras. From scratch, implementation would have been more helpful.

[anshulzade]

This is really nice. Please keep up the good work, the world needs it. If possible can you also share the notebook.
Lots of love from India!

[DezelDanmark]

Why do we need this activation function?

[saikalyangonuguntla]

Your explanation very good keep doing more videos on datascience concepts

[arteigen]

This is great!... This video needs a much high view count. What is going on @youtube???? you need to work on your ranking algorithm.

[donmiguel]

Why does she begins with making the difference between normalization and standardization and then @4:30 describe standardization with mean=0 and var=1 under the name "Batch normalization", does anyone understand this advance ?

[malikfahadsarwar]

so if we have three columns age, weight and height all are in different scale so we don't have to scale them separately instead we can use batch normalization to bring them to same scale