TensorFlow Tutorial #19 Hyper-Parameter Optimization

How to find good hyper-parameters for a Neural Network in TensorFlow and Keras using Bayesian Optimization and Gaussian Processes from scikit-optimize.

This tutorial has been updated to work with TensorFlow 2.1 and possibly later versions.

[azizasaber]

I think this video deserves way more likes than current likes!

[jefferybao]

Thanks so much! I have been searching for how to do grid search with Tensorflow for a while. This tutorial is so well structured and inspiring. Thanks again Hvass for your awesome work!

[kareemjeiroudi]

Wow! You're incredible, well done my friend! Pls keep doing more of these videos. There's a huge audience awaiting for more.

[gabi]

one of the best content i've ever come across

[krean]

This is a great video. Could you explain how to apply this to Machine translation?

[stanislavsmirnov]

Great! An extremely useful tutorial! I was desperately looking for an alternative to a grid search in my project, thank you!
Unfortunately, I could not completely finish it due to error importing 'plot_histogram' and 'plot_objective_2D', error pops up
"cannot import name plot_histogram"; even though i followed your recommendation and installed scikit from a given github link.

[mariembouhadda]

Just AWsome !! Thank you sooo much ! Every thing is running except for the model.load when testing the model, it returns this error : "You are trying to load a weight file containing 5 layers into a model with 0 layers." . I think it has to do with my keras version, i'm using "2.1.6-tf" . online solution proposed adding " input_shape " param but it didn't work .. do you have any idea on how to fix it ? Thank you sooo much again and i'll mention to them how easier things became when using your git repo instead of just using the official release of scikit-optimize !

[nagymahran]

n_calls was set to 40. To me, it is not clear how we can estimate this number.
I believe we need to know:
1. How many samples are required during the exploring phase?
2. How many minimum trials are required during the exploit phase?
adding both should give us the minimum number for n_calls.

[ashwinsingh]

Hi I've been trying this with the mnist.py dataset and using vals such as x_train, y_train_cls, etc. But I get the error: You are passing a target array of shape (55000, 1) while using as loss `categorical_crossentropy`. I think it has to do with the line history = model.fit().


This is what I've put in as input for Keras: history = model.fit(x = data.x_train, y = data.y_train_cls, epochs = 3, batch_size = 128, validation_data = validation_data, callbacks = [callback_log])


I think the problem's mainly with the part: x = data.x_train but I'm not sure what other input to use. Could you release the version using mnist.py?

[lagrenad]

I am getting this error in plot_evaluation() even with dim_names that does not have the categorical dimensions? I am trying with the skopt version that you pointed

[shikunchen]

Hello, I tried to optimize a simple NN with dropout rate, so I just simple followed your code, and add extra lines for dropout rate, but en error arise: fitness() got an unexpected keyword argument 'dropout_rate'
Any idea why? Thank you in advance!

[kareemjeiroudi]

Can I use this fitness function but return auc score instead? Would minimizing the fitness work?
According to my solution it doesn't look so, it's just picking random sets of hyperparameters.

[clau]

Hi,

First of all your tutorials are amazing!!!

I need to do cross validation and I was wondering if instead of using gp_minimize() Can I use BayesSearchCV passing a cv object? Are they equivalent?

Also how I could generate some graphs if I use BayesSearchCV ??

Thanks in advance!!!

[pa]

Keras tuner lets you do Bayesian Optimization

[jhansianumula]

Hi, I am getting below error. Pls help
Cannot interpret feed_dict key as Tensor: Tensor Tensor(flatten_1_input:0, shape=(?, 7, 7, 512), dtype=float32) is not an element of this graph
while running this code:
search_result = gp_minimize(func=fitness,
dimensions=dimensions,
n_calls=40,
acq_func='EI', # Expected Improvement.
x0=default_parameters)
fitness(x=default_parameters) : This code ran fine.
Create_model:
model = Sequential()
train_data.shape[1:])) -- I feel this line may be the reason for error. I am using pre trained VGG16 model without fully connected layer first. Then am using my own fully connected layer.

[WKhan-jlfv]

Hello, Thanks for the tutorial. When I run; It gives an error: TypeError: fitness() got an unexpected keyword argument 'learning_rate' . Any idea how to solve it?
Thanks

[stanislavsmirnov]

Dear Hvass,
As far as I understood your code creates the same amount of nodes on every FC layer, could you please give a hint how your code maybe adapted to meet the requirements of having different amount of nodes in FC layers (i.e. 512 - 256 - 128 - 64 ..etc)

[chainesanbuenaventura]

Is 'plot_histogram' and 'plot_objective_2D' no longer available in the release to the scikit-optimize you made? I am getting this error:

ImportError: cannot import name 'plot_histogram'

[JulianRoinSkovhus]

Did anyone impelent hyperparameter optimization for the LSTM model in tutorial #23?

[maryamparsa]

Amazing tutorial!!! Thanks Hvass!
Thanks