New LLaVA AI explained: GPT-4 VISION's Little Brother

Brand new AI system called LLaVA. LLava 1.5 is a multi-modal system that combines large language models (LLMs) with vision transformers. LLaVA 1.5 is the little brother of our mighty GPT-4 VISION.
The architectural substrate of Lava 1.5 consists of a pre-trained vision encoder and a pre-trained LLM. The vision encoder extracts features from input images, and these features are then used as input to the LLM for generating descriptive or prescriptive text based on user queries. In Lava 1.0, a linear projection layer was used to align the feature spaces of the vision encoder and the LLM, essentially serving as a translator between the two modalities. This straightforward yet effective approach utilizes a trainable projection matrix, which aligns visual features in high-dimensional space to the semantic embeddings used by LLM.

For training and data collection, the authors leveraged GPT-4 to generate a unique multi-modal instruction following dataset. The data set is rich and varied, consisting of three types of instruction-following queries: conversational interactions, detailed object descriptions, and complex reasoning steps. The dataset is amassed from around 160,000 unique language-image instruction-following data samples and is said to outperform even human-generated data in terms of quality and spatial reasoning. GPT-4 was used not only for generating text but also for generating high-quality instruction-following data.

The pre-training and fine-tuning procedure of the Lava model involves two stages. In the first stage, both the vision encoder and the LLM are kept frozen, and only the projection matrix is fine-tuned to align the image features with the word embeddings in the language model. This stage can be seen as optimizing a visual tokenizer compatible with the frozen LLM. In the second stage, an end-to-end fine-tuning process is conducted, where only the vision encoder weights are kept frozen. The trainable parameters of the projection matrix and the LLM are updated, allowing for a more seamless integration between the two.

In addition to the core architecture and training methodology, the Lava model is versatile in its application. It can be fine-tuned to perform tasks ranging from generic conversational queries to scientific Q&A, thereby demonstrating its efficacy as a robust multi-modal AI system. GPT-4 serves a critical role in the AI ecosystem by generating high-quality data that can be used to train other, more specialized models. This reiterates the significance of large language models not just as end-products but as essential components in the broader landscape of AI research and application.

The LLaVA 1.5 model signifies an evolution in multi-modal AI systems, incorporating both sophisticated architecture and an enriched training dataset. One of the most salient features is its integration of the VICUNA 13B large language model (LLM). Given your interest in high-dimensional spaces, it's worth noting that the larger parameter space in VICUNA 13B can capture more complex representations, making it theoretically more capable of understanding intricate, scientific queries. The architectural update of a 2-hidden-layer Perceptron (MLP) as a projection layer between the vision and language models represents another key advance. Unlike the simpler linear projection layer in Lava 1.0, the MLP has the ability to learn more complex transformations between the two different vector spaces. This can be likened to a more expressive function approximator in the manifold of possible feature transformations, facilitating better alignment between vision and language.

The training dataset in LLaVA 1.5 also exhibits noteworthy improvements with the inclusion of a scientifically-oriented Q+A dataset. This is an important addition because standard datasets often lack the complexity and specificity needed for scientific queries.

Another innovation in LLaVA 1.5 is its modularity, which allows for the exchange of either the LLM or the Vision Transformer (ViT). This is a desirable feature from both a research and application standpoint. For research, it allows for the facile investigation into the contribution of each module to the overall system's performance. It provides a more decomposable architecture to study the functional and representational relationships between vision and language modalities. This modularity can be of specific interest for those dealing with graph-based AI systems, as it offers a more granular level of control for implementing alternative graph algorithms, optimization methods, or even layer-wise adaptations.

[carrumar]

Great explanation. Thanks for taking the time

[rishiktiwari]

Really a good summary, are you making a video on training Llama model for action tokens?

[KitcloudkickerJr]

I know LLava is cool but can you imagine they already turned Mistral 7b into a vision model? how does one keep up lol its called Baklava

[idoronen]

I've got a question regarding the LLaVA architecture from the initial paper (LLaVA 1.0, Page 4):

The paper mentions, "...we apply a trainable projection matrix W to convert Zv into language embedding tokens Hq, which have the same dimensionality of the word embedding space in the language model:
Hv =W·Zv, with Zv =g(Xv)."

It seems there might be a typo with Hq being used instead of Hv. Could you clarify what Hq is as referenced in Figure 1? Also, what distinguishes Xq from Hq?

I'd really appreciate an explanation. By the way, the video was excellent. Thank you! :)