Florence 2 - The Best Small VLM Out There?

There is a new VLM on the scene and it comes with a dataset of 5Billion labels. The new model can do a variety of old world tasks like bounding boxes and segmentation along with newer LLM style captioning etc.

For more tutorials on using LLMs and building Agents, check out my Patreon:

🕵️ Interested in building LLM Agents? Fill out the form below

👨‍💻Github:

⏱️Time Stamps:
00:00 Intro
00:13 Florence-2 Paper
02:19 Florence - 2 Architecture
03:20 Florence - 2 Detailed Image Captioning
03:41 Florence - 2 Visual Grounding
04:09 Florence - 2 Dense Region Caption
04:24 Florence - 2 Open Vocab Detection
06:01 Hugging Face Spaces Demo
10:41 Colab Florence - 2 Large Sample Usage

[parkerspitzer]

Thanks for your work on sharing this information. Much easier to watch your content than keep my ear to the ground all day trying to keep up. Much appreciated, sir.

[danielmz]

Thanks for the great content. A video going through the fine-tuning process on this one would be amazing. I am not sure how this could scale to a video implementation (probably passing a frame each time).

[IsxaaqAcademy]

It's also good at OCR for hand written documents

[IanScrivener]

Thanks Sam!!
Please keep up the great work...

[GiovaniFerreiraS]

I'd love seeing a fine tuning video, specially if it's not question answering, just so it's a different use case from the documentation. Maybe with a quick intro talking about what are possible scenarios where fine tune would be specially helpful.

[jefframpe]

Thanks, Sam! I always appreciate your videos.

I would love your take on how Florence-2 compare with Apple's 4M-21.

[unclecode]

This is what people should call "small", anything below 1B! Thanks for your video. By the way, I played around with the quantized version, the result is unbelievably good! I shared a post on Twitter and mentioned you and shared the Colab. Take a look at it. I tried 8 bits and 4 bits. It's odd how 4 bits is almost the same as the base model!

[sohitshivhare]

Thanks for the information this is great.
Can i fine tune it for certain specific images like few short learning. Can you put a tutorial for the same it will be great full.

[ranu]

I've tried this model, describing the image is great. I've also tried the docvqa, but giving only one word answers and not getting even simplest questions right. i had hoped to do some classification and compare with other models.

[RishabhMathur]

@samwitteveenai please make a fine-tuning video about VLMs such as: Llava, Florence-2 and if possible try to use Ollama so that we can make the inference on local device.

[richardobiri]

Thanks a lot for this I wish you could consider the continuing process for identifying authentic and fake certificates 🙏🙏🙏

[yassinebouchoucha]

When will you release a demo on to fine-tune such model ?

[ariramkilowan]

I think fine-tuning for OCR would be a good demo. OCR in the real world with images of documents is much harder than OCR on electronic documents so would be cool to see how a small model like this does as an alternative to Claude/GPT4.

[ShravanKumar]

what would you pick for fine-tuning ?
Any specific application ideas?

[JustEmbraceTheChallenge]

Please do fine-tuning for Object detection

[srk]

We request you to do fune tuning on object detection. Because, all llms are useful generating text oupit only. Thanks in advance

[SinanAkkoyun]

Where is the dataset? I couldn't find the release

[xl]

Is this really idea to use data created by another model to train your model ? 1:54
Isn't it going to replicate the errors from other models ?

[AbhishekKotecha]

Hi Sam, thanks for the video. What do you think about how does it compare with Phi3-V? My take is that this is more raw and better for fine tuning, do you also think so?