Reproducible results with Keras

In this video, we observe how we can achieve reproducible results from an artificial neural network in Keras by setting random seeds for Python, NumPy, and Tensorflow.

🕒🦎 VIDEO SECTIONS 🦎🕒

00:30 Help deeplizard add video timestamps - See example in the description
05:06 Collective Intelligence and the DEEPLIZARD HIVEMIND

💥🦎 DEEPLIZARD COMMUNITY RESOURCES 🦎💥

👋 Hey, we're Chris and Mandy, the creators of deeplizard!

👉 Check out the website for more learning material:

💻 ENROLL TO GET DOWNLOAD ACCESS TO CODE FILES

🧠 Support collective intelligence, join the deeplizard hivemind:

🧠 Use code DEEPLIZARD at checkout to receive 15% off your first Neurohacker order
👉 Use your receipt from Neurohacker to get a discount on deeplizard courses

👀 CHECK OUT OUR VLOG:

❤️🦎 Special thanks to the following polymaths of the deeplizard hivemind:
Tammy
Mano Prime
Ling Li

🚀 Boost collective intelligence by sharing this video on social media!

👀 Follow deeplizard:

🎓 Deep Learning with deeplizard:

🎓 Other Courses:

🛒 Check out products deeplizard recommends on Amazon:

🎵 deeplizard uses music by Kevin MacLeod

❤️ Please use the knowledge gained from deeplizard content for good, not evil.

[deeplizard]

Note, in this video, the variable session_conf is missing. It should be defined before the variable named sess in the following way:
session_conf = tf.ConfigProto(intra_op_parallelism_threads=1,

[Noir_SD]

Yah you just hit the right spot. I wonder why my accuracy changes each time I train my model. Now I got it. So

[rajuthapa]

Finally completed this series. Really helpful videos. I must THANK YOU from my heart. very practical focused and clear explannation. You revise previous tutorials in short, that is one of the best part. Now moving to your deep learning series. I hope to get Tensorflow tutorials from the very beginning like this. Very nice. Thank you Thank you

Will more videos coming in here?? :)

[shimamooo]

With the versions of tensorflow/keras upgraded, you have to make the code compatible with the deprecated functions:


from keras import backend as K


session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)

[prakashjangir]

The way in which you explain each topics i really amazing. I love all your tutorials

[ntd]

If you get error for set_random_seed, change it to "tf.random.set_seed(89)"

[dragosborosgpt]

in the code above session_conf is not defined and creates problems. is there some other code missing ?

[LS-AI]

Excellent videos. Thanks for taking the time to do them.

[Pastamf]

Working code here:


import os
import numpy as np
import tensorflow as tf
from keras import backend as K
import random as rn
rand_seed = 1
os.environ['PYTHONHASHSEED'] = '0'

#random seed for NP genreator of ranodm numbers
np.random.seed(rand_seed)

#random seed generator for Python
rn.seed(rand_seed)

#random seed for tensorflow


session_conf = tf.ConfigProto(intra_op_parallelism_threads=1,
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
K.set_session(sess)

[nerkulec]

I'm so thankful for your devotion for making those videos. They are extremely helpful to start for they allow me to get any results fast while learning. I have huge plans of learning ML and one of my planned projects features a GAN. Can you let me know if you plan on making a video on technically implementing a GAN model in Keras or maybe tell if it's even possible. Anyways, Thank You very much once again!

[goodnine]

😄Finish the whole series video! I feel very happy and I study much useful knowledge, I can't wait to write my own Keras Code!
By The Way
hope the character of your future video can be bigger!

[hellnah]

Thanks for this useful course, I learned a lot from you! :)

[e]

I had this problem the whole day. No such good answer in stackoverflow. But I am watching in bed 🤣 hopefully it will address my problem once and for all

[TheXplorian]

I read somewhere else that pythonhashseed should be set before launching the notebook, as it usually set in the start

[arsheesiddiqui]

{
"question": "Which of the following is not the reason behind not getting reproducible results? ",
"choices": [
"Single thread",
"Mutiple thread",
"Random weight initialization",
"Dropout"
],
"answer": "Single thread",
"creator": "Arshee Siddiqui",
"creationDate": "2020-06-21T11:29:12.451Z"
}

[jamespaladin]

Back on the subject of repeatable results I finally found the answer or at least some of it!
First I looked at the source code for the MobileNet model. I think part of the non-repeatability of the results may occur within MobileNet.Here is the DEF statement for MobileNet:
def MobileNet(input_shape=None,
alpha=1.0, depth_multiplier=1, dropout=1e-3, include_top=True, weights='imagenet',
input_tensor=None,
pooling=None, classes=1000, **kwargs):
It appears that it uses a dropout setting - dropouts are typically a random selection process so I expect for repeatable results dropouts would have to be set to 0. Just tried it, but still the results from two runs are different ! Then I figured it out! The remaining randomness comes from the ImageDataGenerator. Actually it should have been obvious. The call to the imagedatagenerator for the training set and the validation set did not set shuffle=False. Therefore the images produced are shuffled differently in each run and consquently you get different results from the fit generator! I ran the program twice with shuffle=False in both runs. Walla - the results are EXACTLY identical! . Problem is the model will NOT train well because it first sees images of all 1 classification then images of the next classification etc. So you get repeatable results but unfortunately they are not very good results. To get USEFUL repeatable results the random seed for the shuffle in the image data generator needs to be set. In flow from directory you can set the seed. seed= some integer. Did that and ran the program twice. results of that are shown below. UGH - the result are very close but NOT identical like they were with shuffle=False. Have no idea why setting the seed did not produce in identical results. Guess I will just live ith it. Also setting dropout = 0 might cause the model to over train. I checked and there is no parameter to set the seed for dropout in the mobileNet parameters. The third run shows the results when dropout is NOT set to 0 and shows significant variation as expected.
First run dropout = 0 shuffle=True seed=23
Epoch 1/10
- 15s - loss: 0.4148 - acc: 0.8088 - val_loss: 0.1611 - val_acc: 0.9667
Epoch 2/10
- 7s - loss: 0.0959 - acc: 0.9756 - val_loss: 0.1035 - val_acc: 0.9667
Epoch 3/10
- 7s - loss: 0.0395 - acc: 0.9981 - val_loss: 0.1383 - val_acc: 0.9500
Epoch 4/10
- 7s - loss: 0.0206 - acc: 1.0000 - val_loss: 0.0875 - val_acc: 0.9667
Epoch 5/10
- 7s - loss: 0.0159 - acc: 0.9988 - val_loss: 0.0890 - val_acc: 0.9667
Epoch 6/10
- 7s - loss: 0.0105 - acc: 1.0000 - val_loss: 0.0923 - val_acc: 0.9667
Epoch 7/10
- 7s - loss: 0.0070 - acc: 1.0000 - val_loss: 0.0718 - val_acc: 0.9667
Epoch 8/10
- 7s - loss: 0.0053 - acc: 1.0000 - val_loss: 0.0816 - val_acc: 0.9667
Epoch 9/10
- 7s - loss: 0.0036 - acc: 1.0000 - val_loss: 0.0974 - val_acc: 0.9667
Epoch 10/10
- 7s - loss: 0.0030 - acc: 1.0000 - val_loss: 0.0906 - val_acc: 0.9667

Second run same parameters
Epoch 1/10
- 15s - loss: 0.4147 - acc: 0.8081 - val_loss: 0.1620 - val_acc: 0.9667
Epoch 2/10
- 8s - loss: 0.0956 - acc: 0.9750 - val_loss: 0.1038 - val_acc: 0.9667
Epoch 3/10
- 8s - loss: 0.0394 - acc: 0.9981 - val_loss: 0.1385 - val_acc: 0.9500
Epoch 4/10
- 8s - loss: 0.0205 - acc: 1.0000 - val_loss: 0.0880 - val_acc: 0.9667
Epoch 5/10
- 8s - loss: 0.0159 - acc: 0.9988 - val_loss: 0.0883 - val_acc: 0.9667
Epoch 6/10
- 8s - loss: 0.0105 - acc: 1.0000 - val_loss: 0.0918 - val_acc: 0.9667
Epoch 7/10
- 8s - loss: 0.0070 - acc: 1.0000 - val_loss: 0.0704 - val_acc: 0.9667
Epoch 8/10
- 8s - loss: 0.0052 - acc: 1.0000 - val_loss: 0.0806 - val_acc: 0.9667
Epoch 9/10
- 8s - loss: 0.0036 - acc: 1.0000 - val_loss: 0.0952 - val_acc: 0.9667
Epoch 10/10
- 8s - loss: 0.0030 - acc: 1.0000 - val_loss: 0.0887 - val_acc: 0.9667

Third run dropout not set to 0
results vary significantly from runs with dropout=0
Epoch 1/10
- 16s - loss: 0.3965 - acc: 0.8181 - val_loss: 0.1758 - val_acc: 0.9000
Epoch 2/10
- 8s - loss: 0.1092 - acc: 0.9719 - val_loss: 0.1351 - val_acc: 0.9667
Epoch 3/10
- 8s - loss: 0.0455 - acc: 0.9956 - val_loss: 0.1286 - val_acc: 0.9500
Epoch 4/10
- 8s - loss: 0.0230 - acc: 1.0000 - val_loss: 0.0980 - val_acc: 0.9500
Epoch 5/10
- 8s - loss: 0.0159 - acc: 1.0000 - val_loss: 0.0779 - val_acc: 0.9500
Epoch 6/10
- 8s - loss: 0.0099 - acc: 1.0000 - val_loss: 0.0790 - val_acc: 0.9500
Epoch 7/10
- 8s - loss: 0.0064 - acc: 1.0000 - val_loss: 0.0675 - val_acc: 0.9667
Epoch 8/10
- 8s - loss: 0.0046 - acc: 1.0000 - val_loss: 0.0660 - val_acc: 0.9667
Epoch 9/10
- 8s - loss: 0.0037 - acc: 1.0000 - val_loss: 0.0679 - val_acc: 0.9667
Epoch 10/10
- 8s - loss: 0.0029 - acc: 1.0000 - val_loss: 0.0669 - val_acc: 0.9500

[BrettClimb]

I'm guessing that using a single thread will slow down the program substantially?

[jamespaladin]

I implemented the measures to have repeatable results however I found small differences in the results as example below shows
Epoch 1/7
- 20s - loss: 0.6240 - acc: 0.6862 - val_loss: 0.2287 - val_acc: 0.9333
Epoch 2/7
- 10s - loss: 0.2086 - acc: 0.9325 - val_loss: 0.1750 - val_acc: 0.9667
Epoch 3/7
- 10s - loss: 0.1124 - acc: 0.9688 - val_loss: 0.1632 - val_acc: 0.9333
Epoch 4/7
- 10s - loss: 0.0712 - acc: 0.9863 - val_loss: 0.1244 - val_acc: 0.9667
Epoch 5/7
- 10s - loss: 0.0440 - acc: 0.9950 - val_loss: 0.1215 - val_acc: 0.9667
Epoch 6/7
- 10s - loss: 0.0339 - acc: 0.9975 - val_loss: 0.1103 - val_acc: 0.9667
Epoch 7/7
- 10s - loss: 0.0265 - acc: 0.9988 - val_loss: 0.1103 - val_acc: 0.9667

I reset the kernel and retrained the model with these results
Epoch 1/7
- 21s - loss: 0.6239 - acc: 0.6862 - val_loss: 0.2287 - val_acc: 0.9333
Epoch 2/7
- 10s - loss: 0.2190 - acc: 0.9238 - val_loss: 0.1678 - val_acc: 0.9667
Epoch 3/7
- 10s - loss: 0.1026 - acc: 0.9838 - val_loss: 0.1535 - val_acc: 0.9667
Epoch 4/7
- 10s - loss: 0.0678 - acc: 0.9925 - val_loss: 0.1186 - val_acc: 0.9667
Epoch 5/7
- 10s - loss: 0.0404 - acc: 0.9975 - val_loss: 0.1089 - val_acc: 0.9667
Epoch 6/7
- 10s - loss: 0.0319 - acc: 0.9988 - val_loss: 0.0994 - val_acc: 0.9667
Epoch 7/7
- 10s - loss: 0.0242 - acc: 1.0000 - val_loss: 0.1097 - val_acc: 0.9500

The results are pretty close but not identical.I there some other random process at play?

[coolguy-dwjq]

This is was amazing!
I have this question whether to use different weights or imagenet weights when training something specific like hand gesture recognition etc.

[deepaksingh]

Yes, we do find evrything helpful ij videos. Made. By yoh.. Please keep continuing making videos on small topics or. Subsections as well..

Als. It will be great if you could add videos on RNN, Lstm, GAn etc. In the machines learning series.
Those are also kind of essentials Nowdays..
Although i already ready those in books but still nothing can clear concepts like ur videos do:)
So. Please do that as well..