Use the OpenAI API to call Mistral, Llama, and other LLMs (works with local AND serverless models)

An incredibly easy way to call local or hosted models using the exact same OpenAI API (or LangChain).

Now that Ollama is compatible with the OpenAI API, we can use the same API to call either local models via Ollama or hosted models via most of the serverless LLM providers.

Interested in talking about a project? Reach out!

Timestamps:
0:00 - Intro
0:38 - How It Works
1:38 - Code Demo - Tool Calling
3:14 - Code Demo - Streaming
4:22 - Code Demo - Swap Between 5 Models
5:39 - Code Demo - LangChain Compatibility
6:18 - Code Demo - Agent Comparison
10:52 - Benefits of Interchangeability

[HarpaAI]

🎯 Key Takeaways for quick navigation:

00:00 *🧩 Swapping Between Local and Serverless Models*
- Easily switch between local LLMs and serverless models using the OpenAI API.
- Challenges and considerations when using serverless options like AMA for open AI compatibility.
- Demonstrating how to toggle and set up models for different requests effortlessly.
03:37 *🚀 Utilizing Various Models in Tool Calling*
- Exploring the models compatible with tool calling in serverless options like Together.
- Highlighting the importance of model support for tool calling functionalities.
- Demonstrating how to effectively run requests through different models and analyze their responses.
08:25 *💻 Optimizing the Model Selection Process*
- Comparing the efficiency of different models for tool calling, including MRA, M, and Mistral.
- Discussing the flexibility of interchanging models based on performance and cost factors.
- Sharing insights on the future landscape of model swapping and the competitive advantages of serverless models.

Made with HARPA AI

[runvnc]

One thing that is misleading in your video is that you make it sound like you can swap out just about any model for tool calling. That is false. Mixtral and Codellama are smart that way and can handle it. Many models will not be able to do it, or not reliably enough.