Optimizing Databricks LLM Pipelines with DSPy

In October 2023, researchers working in Databricks co-founder Matei Zaharia’s Stanford research lab released DSPy, a library for compiling declarative language model calls into self-improving pipelines. The key component of DSPy is self-improving pipelines. In a complex, multi-stage LLM pipeline, there are often multiple prompts along the way that require tuning. We discuss a blog post that dives into how to build a custom, multi-tool LLM agent using readily available Databricks Marketplace models in DSPy and how to deploy the resulting chain to Databricks Model Serving.