Building an AI API Gateway

Since everyone is putting "AI" on everything, let's also put it on the API gateway!

I don't necessarily like the term "AI API Gateway," but it sounds cool. When thinking about "AI" features in an API gateway, I'd put them into two buckets:

1. Using API gateway to manage LLM APIs

This includes any "traditional" API gateway features like authentication, authorization, rate limiting, and request/response/header modifications, but applied to the LLM APIs. For example, augmenting the user prompts, allowing or disallowing specific prompts, and so on.

2. Using LLM functionality in the API gateway

This bucket contains features where the API gateway uses an LLM to do something. For example, you could use AI to detect anomalies in the traffic or to detect and block malicious requests. You could also use LLM to generate responses to specific requests or augment the responses coming from the backend.

As I like to try things out, I took a proxy I know - Envoy - and started implementing some of these features using a WASM plugin.

In this stream, I'll show you what I have worked on so far and will try to implement some more features.

If you like the video, consider subscribing:

▬▬▬▬▬▬ Connect with me ▬▬▬▬▬▬

[kaushikmitra-vf]

Can you build a custom queue for the requests to schedule them based on priority before routing them to backends