Mastering LLM Delivery in Private Clouds: A Journey to Seamless Dep... Autumn Moulder & Marwan Ahmed

Mastering LLM Delivery in Private Clouds: A Journey to Seamless Deployments with Kubernetes and OCI - Autumn Moulder & Marwan Ahmed, Cohere

Deploying LLMs is challenging. This talk is a case study in how cloud native technologies, specifically Kubernetes and OCI artifacts, simplifies private LLM deployments. Allowing teams to run models in their infrastructure solves significant data governance & security challenges. However, it is still difficult to efficiently share large artifacts between model developers and model consumers. Autumn and Marwan share how open standards unblocked challenges and simplified LLM delivery. First, we explore how Kubernetes made it possible to rapidly deliver a highly portable, cloud-native inference stack. Second, OCI Artifacts have been underutilized as a delivery mechanism for artifacts beyond container images. We explore how we achieved significant efficiency gains by reducing duplicate storage, increasing download speed, and minimizing governance overhead. Walk away learning how you can leverage Kubernetes and OCI in your MLOps journey.