801: Merged LLMs Are Smaller And More Capable — with Arcee AI's Mark McQuade and Charles Goddard

#LLMs #ModelMerging #SmallLanguageModels

Merged LLMs are the future, and we’re exploring how with Mark McQuade and Charles Goddard from Arcee AI on this episode with @JonKrohnLearns. Learn how to combine multiple LLMs without adding bulk, train more efficiently, and dive into different expert approaches. Discover how smaller models can outperform larger ones and leverage open-source projects for big enterprise wins. This episode is packed with must-know insights for data scientists and ML engineers. Don’t miss out!

In this episode you will learn:
• [00:00:00] Introduction
• [00:02:55] Explanation of Charles' job title: Chief of Frontier Research
• [00:03:10] Model Merging Technology combining multiple LLMs without increasing size
• [00:13:31] Using MergeKit for model merging
• [00:21:21] Evolutionary Model Merging using evolutionary algorithms
• [00:26:28] Commercial applications and success stories
• [00:36:31] Comparison of Mixture of Experts (MoE) vs. Mixture of Agents
• [00:52:33] Spectrum Project for efficient training by targeting specific modules
• [00:59:50] Future of Small Language Models (SLMs) and their advantages