Learn Streaming SQL in 3 min, e6: Streaming ETL, how to mask PII data with Proton and Streaming SQL

You can apply streaming ETL on your live data, just with SQL.

In this 3min video, Jove showed how to avoid sharing the raw IP address to your data engineers/analysts, by apply md5 on them. We will read live data from a Kafka topic, apply the transformation on ip field and write the masked data together with other data in the other Kafka topic. The key SQL is

CREATE MATERIALIZED VIEW mv INTO target AS
SELECT now64() AS _tp_time,
raw:requestedUrl AS url,
raw:method AS method,
lower(hex(md5(raw:ipAddress))) AS ip
FROM frontend_events;