Using OpenAI Realtime API to build a Twilio Voice AI assistant with Python

Chapters:
0:49 Prerequisites
1:48 Code starts here
2:40 Configuring the application
3:42 Handling the incoming call
4:26 Setting up websockets
5:10 Handling Twilio Media Stream events
5:53 Handling OpenAI Realtime API events
7:11 Configuring send_session object
8:16 Starting the application
8:55 Configuring our Twilio phone number
9:20 Testing the application

This video will help you build an AI assistant using Twilio Voice and OpenAI's Realtime API. Here's what you'll need to build it:

Resources:

[riley_blackwell]

Thank you for creating this demo in python! As I am building a startup, I have come to realize how essential the Twilio platform is. 1. You need Stripe to collect payments from users 2. You need user authentication such as Oauth 3. You need to be able to communicate with your users and collect data about your users. That is where Twilio is essential with its communications apis and segment to collect user data and build profiles. Keep up the great work, and keep investing in the core communications and data products. 😊

[AITeam-iq]

Thanks @TwilioDevs for making such an amazing video. It's helping us a lot.

[Avextom-ozsv]

This was cool!
My requirement is to build the voice feature on web based chat system in addition to Twilio phone call. I hope I will do it with this video

[rfouyang-simai]

It seams like this code cannot work using websockets version 14. can only use websockets 13

[herbyprein]

Perhaps you could also make a video with a voice no-code/low-code application (e.g. with N8N). For example, you call the Twilio phone number, send a webhook to N8N to start a workflow, perform actions there based on the call (e.g. query a database, RAG, ask an AI, ...) and return the answer. Most videos you find on Youtube just call another service that handles the call.

[bakhtullah]

Have you implement your own call flow instead of getting response from openai?

[in-my-opinion]

Hi, how could we scale this application when 1000s if users start calling concurrently

[zivgamer]

Is there a way to trigger the first response without needing to say something first?

[adityabarman]

Can we call our free trial twilio number from any verified number globally once the url is added to the voice config webhook url

[physicsmadness]

Do i need to purchase a number or this works even in the trial sample number that they provide?

[EntertainmentDoseByAkash]

How can we integrate the knowledge base to this so that our voice assistant can answer referring to a given data source?

[GaryShirk-vr]

Hi is there anyway I could buy this system and if someone was to buy a system like this how would you go about it I don't have the time to build it I already spent $3000 having someone build it and it did not materialize

[gioegiogcbr]

Great video! Can one interrupt the AI assistant while is talking?

[mattthorn]

Thank you for making this great example, it's been invaluable. Just one question: I'd like to have the Agent run a custom prompt defined at runtime, so I need to pass a parameter into this call: await But I cannot find a way to pass a parameter into the media-stream endpoint and access it in time to pass to initialize_session. I've tried a custom param, but cannot seem to retrieve it, and querystring also doesn't seem to work. Would you have any guidance on this. Perhaps using Conversation Relay is the answer? Thanks for your help.

[bulutosman]

The repo in the description and the code in the video does not match

[rohitmane]

Can you give me suggestions on how I improve my voice to text problem of twilio, I used gather function but they don't take right input or it cannot correct transcription?

[Anything-ebwf]

I am deploying this thing in gcloud and trying to call the webhook through public url and calling the webhooks through call initiation . Up here it's saying the initial things but call is getting disconnected and saying websocket refusal error. Is there anybody in your team who could help me with this?

[FintanDeveloper]

is it possible to make a call on my given number with this setup ?

[hakansuzer]

Is it possible for the agent to respond based on our documentation? For some topics I want it to answer according to my context and the information in my documentation, how can I do it?

[bakhtullah]

If anyone want to make an outbound call. does the system will work for it?