How To Fine-tune LLaVA Model (From Your Laptop!)

In this guide, we fine tune the popular open sourced model, LLaVA (Large Language-and-Vision Assistant) on a dataset to be used in a visual classification application. You can perform the fine tuning yourself, regardless your level of experience, or the level of compute you have access to.

Please leave any future guides you would like made below!

[RehanKhan-pstn]

You know bro is a A level engineer when he can explain stuff soo easily

[TomanswerAi]

Best guide/insights on fine tuning I’ve seen. Subscribed 🔥

[ae_alg]

Baxate, you’re the goat. For a beginner like myself, that was a very useful video

[hubertboguski]

I tried out brev for my machine learning course, love the options for the payment system where i have the option to cut if off after X dollars, low prices, and ui looks awesome. I know it said it somewhere but it took me a minute to realize that my Jupyter notebook takes around 4 minutes to launch so for blind ppl like me I’d put some more text saying Jupyter notebook will be created in X minutes.

Love this vid and outreach- I’ll keep watching Baxate

[atriantafy]

Great video:) Can you please comment on the dataset size? The one you used consists of roughly 9k samples. How many samples are needed to have a decent lora fine-tune? I've heard that with LLMs you can achieve much even with only a few examples. Is it the case for LLava as well? Please share any more information you can on the dataset creation. thanks!

[shivanshsingh]

Hello bro, after running the deepspeed script there is no file with name mm_projector.bin is generated which is required in merging process but a non_lora_trainable.bin is generated

[madhavparikh]

Hey, I had a query regarding generating the custom dataset using gpt 4, shown at the very beginning. It seems it does not generate json file with the exact format necessary for LLaVA

[paulmiller]

Cool demo, thank you. Could you share some examples of training data? That new model is great. Can you share it on Hugingface? How big did it end up being for inference purposes,

[freddyfly]

is it possible that the model can tell you there a picture was taken(geographic), based on probability, and purely focuses on this, because you give him the information in fintune( im a beginner)

[zwdoumr]

Have you used this link? I'm reporting an error when loading the dataset now, if you can please take a look . thank you

[drsamhuygens]

For this use case, why didn't you just use prompt engineering (using a very specific prompt) to give you the same output?

[tysonla]

Do I have to buy credits to follow along?

[Snorlaxer]

Came from TikTok! But I have no experience w AIs but am surely going to dive in to train a model for my startup application. Do you think this model could be trained to estimate macros from an image, let’s say in buckets or ranges, after identifying the food itself?

[aimattant]

Wouldn't prompting the LLM in various scenarios in the application code be enough to get the right response? I am not clear on fine-tuning.

[raresracoceanu]

Can you show how to fine-tune VILA models from Nvidia?

[kukiui]

i see you're finetuning LLaVA 1.5 is it possible to use this notebook for 1.6 too?

[BR-lxpy]

Wouldn't it have been simpler to feed the fluffy text to llama3 to come up with the summary?

[aamirshaikh]

In what world is this a “beginner friendly machine learning guide”? What💀💀💀😂😂