How to Deploy ML Models in Production with BentoML

In this video, you can learn how to deploy Machine Learning models into production using BentoML. I explain how to install BentoML, how to save ML models into BentoML's local store, how to create a BentoML service, how to build a bento, and how to containerise a bento with Docker. I also send requests to the BentoML service to receive back inferences.

Check code:

=================

Join The Sound Of AI Slack community:

Interested in hiring me as a consultant/freelancer?

Connect with Valerio on Linkedin:

Follow Valerio on Facebook:

Follow Valerio on Twitter:

=================

Content

0:00 Intro
0:19 BentoML deployment steps
1:03 Installing BentoML and other requirements
2:11 Training a simple ConvNet model on MNIST
5:52 Saving Keras model to BentoML local store
10:26 Creating BentoML service
15:28 Sending requests to BentoML service
22:06 Creating a bento
25:46 Serving a model through a bento
28:25 Dockerise a bento
30:46 Run BentoML service via Docker
32:41 Deployment options: Kubernetes + Cloud
33:43 Outro

[TridomML]

Would love to see an update to this video, perhaps with a more complex model framework such as PyTorch or Huggingface!

BentoML has updated some of its infrastructure to handle custom objects, which can store the tokenizer for these models, but there's a gap in resources that demonstrate how to utilize that tokenizer in the `predict()` service function definition.

[ThomasFackrell]

This is super cool! I can't wait to share it with my team.

[nguyenthehoang]

Thank you for your time and effort in making this video!

May I have some comments:
1) The good:
- The content is great
- About the video itself, everything is great (e.g., light, speech, clarity of explanation, .. and so on)

2) The not really good:
- People, looking for solutions for our showcase projects, don't see what we are looking for in the video and probably go back to some tutorials about using Flask to serve our models. Here's why: we probably want to make a web app or a mobile app (e.g., using react native) and include the link to the resume. Your content isn't finished, so it isn't convincing enough.

[vivkpij]

Thank you for the tutorial, I have a question, in my project I have multiple edge devices, can I use BentoML's runner instance to run on multiple nodes?

[shadyizloo]

hey i need to know how does python work when you have two separate train and test files. like we train on one file and then test on another csv file. please guide if you know how does that work

[marinamaayah]

I had this error How can I chck the server log to know what exactly the error
rediction: "An error has occurred in BentoML user code when handling this request, find the error details in server logs"

[aibdarija]

Can you do the blind source separation

[Diegownz]

Why did you make a json file of the input?

[jawadmansoor]

How do you deploy voice recognition system (as in your previous videos) on bentos?
Here at 13:39 on line 16 input and output is numpyNDarray, what would it be for audio data?

[manhpham]

Dear sir, can I ask why should we use BentoML, while there're loads of robust serving frameworks? Or in other words, can you compare BentoML with others... Thank you!

[sarthak]

I am not able to containerize using bentoml..

[iva]

I'm stuck: from mldeployment import training
ImportError: cannot import name 'training' from 'mldeployment' (unknown location)

[haoshoku]

Hello I I have a question I want to make a project on removing stutter from a speech signal what data structures or tools do I need in order to make this?

[shubhamtalks]

To be very honest, BentoML doesn't seem to be adding any value! Why would anyone switch from FastAPI to this?