TensorFlow Tutorial #18 TFRecords & Dataset API

How to use TFRecords with the Dataset and Estimator APIs in TensorFlow.

This tutorial does NOT work with TensorFlow 2 and later versions, and it would take too much time and effort to update it.

[strachol]

I have managed to get 78% accuracy on 'test set' pictures (on the 'train set' the loss hovers around 0.02-0.002) after some tweaks to the actual files in datasets.
I changed them so those are more equal in numbers. (there were too much spooneys).
Also I shufled the pictures in test and train folders so those are more uniform and not that test folder is full of cutlery on grass as opposed to more abstract backgrounds in the train folders.
(This was using CNN approach) the end spread of predictions was very uniform 82, 102, 107 and with 78% accuracy I think it is preety good score on such 'hard' dataset. So the code is definitelly on the good side.

Also people, please remember that after trying different approaches you must actually restart your model (delete chackpoints) bacause if it is already overfitted (by past training sessions with bad approaches) then trying to fix it is nearly impossible.
It's better to start with clean model and then when results are better then continue on the same model by doing smaller tweaks.

I learned so much from this tutorial especially by experimenting. Thank you very much! It was awesome.

[Kaizala]

Thanks dude, working with tf.Data has jus been a pain in the ass for the past 4 days...

[richardf]

What a fantastic tutorial!!! Informative and well organized thank you so much

[BeGunNer]

7:26 HAhaha so much saltiness Hvass. I love this video, but I started out laughing to a bunch of code in on a screen in a cafe. This does not make me look like a normal person.

[noli-timere-crede-tantum]

I wish code reviews at work were done in this fashion. Painful, but instructive

[bendesign]

Very useful tutorial thanks again! Can you omit the Estimator API and use the TFRecords on their own?

[youmustbenewhereguy]

Hi, nice tutorial. A question tho, If I have 2D array float as an input, not an image, do I have to .tostring() and wrap it in bytes too?

doing that somehow returning different size of array when parsing it back.

[mihailmihaylov]

Hi!
I have followed your tutorials and they are all extremely helpful. These are the only good tutorials I have found. One question though, would you make an update on how to load tfrecords with tfdataset and a customer estimator for the "beloved" tensorflow 2 ? I have been struggling for quite some time now, and I cannot find any reasonable documentation whatsoever. Thanks in advance!

[weichengzhu]

Hi Hvass, thank you for your nice tutorials. Any suggestion or tutorial on using estimator and dataset API for tensorflow serving? Thanks

[sam]

many thanks for uploading this. Just a quick note: I have a task where I need to resize the images before making tfrecords, I used cv2 for resizing followed by 'img.tostring()' but when I saw the size of tfrecords file, it was quite large, I tried using python's native file open to store the image in tfrecord without downsizing and the resultant size of tfrecord file was much lower. Any comments on that?

[malekibrahim]

Can you still use these functions if your data labels are not one-hot encoded?

[blackrachmaninov]

Very nice video ! How to deal with a label which is a list and not a single value ?

[PavanKumar-jcqn]

Hi,
This is an Excellent presentation. After watching your video I am trying to run another deep learning program that uses estimators written in tensorflow but the gpu utilization is stuck at 8%. How can I improve the GPU utilization. Any pointers?
Thank you
Regards,
Pavan

[christianherz]

I felt the same way! 7:27 really made me laugh.

[timephonic]

Hi Magnus, Your tutorials are very helpful. I was wondering can you make a tutorial on handling multilabel image classification in tensorflow?

[Salmariazi]

Thank you so much for this tutorial! Tensorflow documentation is horrible!

[balajichetty]

Hi,
Nice tutorial.
In my case, I am using model.fit method.
And I am getting the following error.
TypeError: Unrecognized keyword arguments: {'input_fn': <function train_input_fn at 0x7f6e91ef1d08>, 'steps': 200}
help me how to use in model.fit
Thanks

[kislaykunal]

Ok, so if anybody had access denied you have to specify a file name for the records, not just a directory name.

[syedjameer]

Hello Hvass Laboratories Can please make a video session image captioning . It is really worth of learning and I have learned CNN from your tutorials.But when move to image captioning . It is combination of ENCODER AND DECODER. So could you please make a video session on image captioning. It would be helpful

[EdBordin]

The reason the API is so hard to use is that it comes straight out of the protobuf compiler... they should have wrapped it in a higher level API.