Deploy a custom model to Vertex AI

Steps to import a Keras model trained in Colab into Vertex AI, deploy the model to an endpoint, and validate the deployment.

Thanks to Codence for the music.

[AmitKumar-hmgx]

Hello Mark,

first of all, thank you for the video, it's explained in a very simplistic manner. Again, you have replied to almost all the comments something I have hardly witnessed before, well done!

I am moving from Azure to VertexAI and quite frankly I know nothing about the latter.

I have a few questions:

a) to register a model in model registry, is it a mandate to first upload the files in the bucket ?
b) Now, I understand if it's a BERT based model that I have fine-tuned, I can upload the model files but what I am using a LLM for inferencing and I want the model to be used as an endpoint 'Vertex AI endpoint' can I do that ?

Basically my goal is to register diff models for each module in the Vertex AI and then adapt my containers to use those models which I believe in this case would be to use as an endpoint service.

Any help would really be appreciated.

[MeynoushKOHANDANI]

Thank you so much! I tried that and it worked for me. Now I have to try it with my own model. Thank you so much Mark.
@theMeynoush

[Goutamchap]

wat format shud the uploaded model be to upload it into bucket and deploy? I have a python file running one of the hugging face transformers, how do i upload that into vertex ai?

[comrade_pasta]

Hey Mark, thanks for the video. I have the model imported and running on an endpoint, but I’m seemingly not able to run it in the Chat module within vertex. Any suggestion for what I might be doing wrong?

[anwarmujeeb]

if there is data processing required prior to feeding into the model (such as scaling) how do we import the preprocessor?

[RishavJayswal-yzdh]

Hey! Can you show us how to deploy our model and enable a grpc endpoint? I want our live services to talk to the model for inference using grpc.

[gilangthehuman]

Hello Mr.Mark

I dont have much experience with this, but i am trying to learn.
After deploying the model to Vertex AI, how can we call the model to be used on a mobile app?
We will be using API right for that correct?
thank you for the video

[shubham_chime]

Hi Mark, nice video. I have a question. Once the model the deployed, does Vertex AI save the input JSON request and response object anywhere? If yes, how can one access it?

[whitetiger]

Did you need to grant some special permissions to the service account in order for the deployment process to succeed?

[cryptotechinfo]

Hi Mark,
Thanks for video. can you guide me how to develop Q&A ML model using vertex AI

[SurchandW]

How do you save the model with the output shape defined in tensorflow? The model, I deployed throws {
"error": "Tensor name: prediction has no shape information "
}. While saving the model, I have defined the output shape as:
from typing import Dict
class ExportTranslator(tf.Module):
def __init__(self, translator):
self.translator = translator

@tf.function(input_signature=[tf.TensorSpec(shape=[1, ], dtype=tf.string)])
def __call__(self, sentence) -> Dict[str, str]:
(result, tokens, attention_weights) = self.translator(sentence, max_length=MAX_TOKENS)

return {"prediction": result}

when I check the output using saved_model_cli, it gives:
The given SavedModel SignatureDef contains the following input(s):
inputs['sentence'] tensor_info:
dtype: DT_STRING
shape: (1)
name: serving_default_sentence:0
The given SavedModel SignatureDef contains the following output(s):
outputs['prediction'] tensor_info:
dtype: DT_STRING
shape: ()
name: StatefulPartitionedCall_2:0
Method name is: tensorflow/serving/predict

Any idea on how to fix the issue?

[otisrohman]

So, can the public test the model with the endpoint?

[aotrakstar]

Hi Mark, thanks for the video. Are will this same method work for a yolov8 project? say i upload the entire projet files are the model?