Ray Serve: Patterns of ML Models in Production

(Simon Mo, Anyscale)

You trained a ML model, now what? The model needs to be deployed for online serving and offline processing. This talk walks through the journey of deploying your ML models in production. I will cover common deployment patterns backed by concrete use cases which are drawn from 100+ user interviews for Ray and Ray Serve. Lastly, I will cover how we built Ray Serve, a scalable model serving framework, from these learnings.

[dinoscheidt]

I’d like to see an End-To-End hands on example of this. I.e. as a coding tutorial or such which shows hands on how to do large time series analysis / large model and carry it to production. Simply cases were you need to train on many machines / GPUs to get through the data (see NVTabular) and how ray clicks into that regarding the research, training, validation (weights and biases / tensor board) and finally serving / inference phase. Right now ray feels like an “take all or nothing” compared to RAPIDs etc.