How to train a model to generate image embeddings from scratch

Embeddings are one of the fundamental building blocks behind Large Language Models.

I built a simple model to generate image embeddings. This video will help you understand embeddings from first principles. I don’t use transformers or anything fancy. Instead, I build a simple Siamese Network step by step, and train it using contrastive loss.

I teach a live, interactive program that'll help you build production-ready Machine Learning systems from the ground up. Check it out here:

To keep up with my content:

[emrahe]

I had been working on a similar problem for a few weeks and had already implemented most of the code you mentioned (after many trial and errors) . But after watching your video, I realized that I had missed a few crucial details like the dense layer and the loss function. Your clear instructions and fantastic tutorial really saved me tons of of time. I wish you had released this video earlier, but regardless, thank you very much! 🙏

[LuisAlvarado-hmbr]

Great, insightful video with an original approach to explaining embeddings. Most explanations focus on text, so it's refreshing to see image embeddings for a change. It's also fantastic to see such an influential paper used as a reference for the implementation. Thank you!

[toddroloff]

Great video. I like your enthusiasm, and passion you display in your videos. The way you break things down and explain it is great. Thank you

[ThetaPhiPsi]

Contrastive explained nicely! It's a shame nobody uses it.

I've some improvements to add:
1. you can use the model itself to compare pairs and take the loss to discriminate results (but the embedding is fine too for a class of downstream tasks)
2. you can further take ROC AUC and optimize your threshold on the given training data (I used sigmoid to squish the loss between 0 and 1)

Works nicely!

[chidubem]

cool explanation, i always wondered how embeddings worked at the lower level

[ojaspatil]

thank you for the intuitive explaination!

[KoenYskout]

I experimented with modifying the embedding size to 2, and visualize that on a 2d plot (colored by label). Easy to see how all (or most) numbers with the same label are clustered together by the embedding, and numbers with a different label are moved apart.

[gemini_]

Gemini 1.5 Pro: This video is about creating image embeddings from scratch using a neural network.

The speaker starts by explaining what embeddings are and why they are important. Embeddings are a way of representing data points as vectors in a high-dimensional space. Similar data points will have similar embeddings, while dissimilar data points will have dissimilar embeddings. This makes embeddings useful for tasks such as finding similar documents or images.

The speaker then introduces the concept of a Siamese network. A Siamese network is a type of neural network that takes two inputs and outputs a measure of similarity between the inputs. The speaker explains how to use a Siamese network to train a model to generate image embeddings.

The speaker then shows how to train the model on a dataset of handwritten digits. The model learns to generate embeddings for the digits such that similar digits (e.g., two different images of the digit 3) have similar embeddings, while dissimilar digits (e.g., an image of 3 and an image of 7) have dissimilar embeddings.

Finally, the speaker shows how to use the trained model to generate embeddings for new images. The speaker concludes by discussing some of the applications of image embeddings.

[yaseromar]

What a magnificent explanation, every time I watch one of your videos I feel enjoyment and excitement and I can see the same in your way of talking about machine learning 🤩🤩🤩🤩🤩🤩🤩🤩🤩🤩

[sachinmohanty]

Beautiful explanation ❤ loved the tutorial 😊

[Aclodius]

You're doing the Lord's work

[sam.scrolls]

Thank you for the wonderful explanation. I understood the importance of loss function here. If I want to create an embedding with multiple objects in one image, can you please give some insights on how it can be done?

[LanreOladele]

I sincerely would like to see how you'd go about it using 3d images while implementing triplet loss

[LanreOladele]

@Underfitted, Thank you for this amazing video. How would you ideally do the same using 3d images?

[ian-haggerty]

<3 u Underfitted! This couldn't be more timely for me :D

[chuanana]

Thank you for the video! Is it expected to have the distance of image embeddings of different labels (3 vs. 7) to be greater than 1? I got (1.0468788, 1.087123). Since we normalized the inputs, I had expected the embedding distance to be normalized as well. Is there an expected range for the distance?

[ddemmkkimm]

1:51 Image is not 2D data. It is # of pixels dimensional data, i. e. width x height.

[thevoyager]

Thanks for the nice explanation!
Could we use these image embeddings for classification tasks? if so, how?

[wmxrbzb]

Thanks for the video! so am i right to say that the process is the supervised learning?

[ian-haggerty]

Funny, it wasn't too long ago that MNIST wasn't a "toy" problem. The history of computer vision is rather short. Are we writing the beginning of it?