The Wrong Batch Size Will Ruin Your Model

How do different batch sizes influence the training process of neural networks using gradient descent?

📚 My 3 favorite Machine Learning books:

Disclaimer: Some of the links included in this description are affiliate links where I'll earn a small commission if you purchase something. There's no cost to you.

[ErlendDavidson]

If you scale the batch size by the learning rate (i.e. lr=(batch_size/32.)*0.01) then the stochastic gradient descent looks sort of okay here.

[Metryk]

Hi! Maybe you can help me with this one: if I want to test an already pre-trained image classifier, how do I proceed regarding the amount of images used? The set containing test images has 100k images, I guess it wouldn't make any sense to load them all at once, so how do I proceed? Thanks!

[johnmoustakas]

Good work, hope your channel gets more attention

[ErlendDavidson]

What do you think of (artificially) adding noise to the learning rate. I feel like it used to be more popular to do that, but almost never see it these days.

[Agrover]

Hey love this video! Was losing touch of the basics !

[muhammadtalmeez]

Your videos are amazing. Thank you so much for this great knowledge and beautiful videos.

[zrmgwci]

i didnt see a helpful video like this one in the entire internet, thank you ♥

[sarahpeterson]

the question is whether if you use a batch and reach the global minimum is your model functionally equivalent to one that didn't batch? Are the weights identical... no they aren't . if your model is generative you don't have equivalence with batch/non batch.

[axelanderson]

If you generate a dummy dataset and set a static learning rate, then smaller batch sizes work better? wtf?

[OliverHennhoefer]

Really like the videos. However, I want to warn against the general statement that a batch size of one is not recommended. It really depends on the problem/data. So don't simply dismiss stochastic gradient descent, try it!

[ziquaftynny]

I love your presentation style! Very energetic :)

[edmundfreeman]

This is the kind of thing that I hate about deep learning. A single parameter in the optimization method can completely change the results. Batches should be small but not too small. How small? That's for heuristics but will change on different data sets.

[Levy]

Amazing!!

Did u know why the batch size os always 32, 64, 128?

[akshay]

Good content. Try improving ur way of teaching. Learning should in relaxed tone