Hyperparameter Optimization - The Math of Intelligence #7

Hyperparameters are the magic numbers of machine learning. We're going to learn how to find them in a more intelligent way than just trial-and-error. We'll go over grid search, random search, and Bayesian Optimization. I'll also cover the difference between Bayesian and Frequentist probability.

Noah's Winning Code:

Hammad's Runner-up Code:

More learning resources:

Join us in the Wizards Slack channel:

And please support me on Patreon:

Thanks to Veritasium (bayesian animation) & Angela Schoellig (drone clip)
Follow me:
Signup for my newsletter for exciting updates in the field of AI:

[surajthapa]

You are one of the rare educators who can make smile their viewers in between learning which makes learning flowless. I believe without any stop I can watch your 1 hr long content too. Thanks for making learning easy and funny.

[JulianHarris]

Holy crap I can say with confidence this is the funniest introduction to hyperparameter optimisation there will ever be. Ever. Genius work. You don't call any more, but that's ok. Live your live, enjoy it! Be free! Be yourself!

[DaredevilGotU]

Every time I visit the page, I learn a new technique. Thanks Siraj

[tumul]

dude u making learning so awesome !! great work

[peretz]

Your geeky / cringey jokes are the best! Don't stop. Seriously.

[Skythedragon]

Yo dang, I heard you like optimizers, so I made an optimizer to optimize your optimizer

[dexterdev]

Thanks for the effort you are taking for these videos. I respect it. :)

[powerrabbit]

This is the coolest channel on Youtube!

[swannabesaliha]

Pretty cool video...good job Siraj.. thankyou...

[phillipotey]

Just figured something out with nodes. Length amount of nodes is cleverness, height of nodes is smartness.

[rajatmaheshwari]

Et u Brute Force... I laughed so hard at this point.

[elsmith]

This is amazing! Thanks for the video :)

bayesian optimization itself has hyperparameters, like kernel window size for gaussian process fitting.... all these BO libraries have these parameters at some default value, which almost always does not work for your model at hand...

[FinanceLogic]

Bayes is not as random as it seems you think around 5 minutes in. But I did learn a lot here. thanks.

[Ninja-iqxt]

Loved this pace, atleast it makes us understand whats going on, as compared to the previous vidoes, which are quantity over quality.

[phil.s]

Thank you for this video, i am currently testing Scikit Optimize to optimize the network i am currently working on.
It supports Bayesian optimization and is simple to implement where as hyperas likes to give errors.

[randompast]

Good explanation, illuminated a few things for me, thank you.

[FinanceLogic]

It just clicked how a random forest really works less than 1 minute into this video. i feel sick because the world is so interesting.

[_jiwi]

Hi Siraj, thanks tons for the video! I am unsure of what you meant by utility of the expectation of function f. You said it tells us which region of domain of f are best to sample from, but I can't quite follow what you mean by that. Would highly appreciate some help with this!

[justinwhite]

Here I was thinking early in the video 'would a Monte Carlo approach work?' when you got into talking about exploration /exploitation I think it might.

This higher level math you are doing here I don't get (or maybe I need someone else to explain it) but Monte Carlo is something I've used before and I think it might be good enough.

You could seed in a set of likely values and let it add new ones when it heads to an upper or lower bound. The nice thing about Monte Carlo is that it would explore possibilities as the model matures and switch over to something if it winds up performing better.

This obviously works better for integer parameters than for gradual values.