Bringing Pure Python to Apache Kafka (with Tomáš Neubauer)

The “big data infrastructure” world is dominated by Java, but the data-analysis world is dominated by Python. So if you need to analyse and process huge amounts of data, chances are you’re in for a less-than-ideal time. The impedance mismatch will probably make your life hard somehow.

So there are a lot of projects and companies trying to solve that problem. To bridge those two worlds seamlessly, and many of the popular solutions see SQL as the glue. But this week we’re going to look at another solution - ignore Java, treat Kafka as a protocol, and build up all the infrastructure tools you need with a pure Python library. It’s a lot of work, but in theory it would make Python the one language for data storage, analysis and processing, at scale. Tempting, but is it feasible?

Joining me to discuss the pros, cons, and massive scope of that approach is Tomáš Neubauer. He started off doing real time data analysis for the Maclaren’s F1 team, and is now deep in the Python mines effectively rewriting Kafka Streams in Python. But how? How much work is actually involved in porting those ideas to Python-land, and how do you even get started? And perhaps most fundamental of all - even if you succeed, will that be enough to make the job easy, or will you still have to scale the mountain of teaching people how to use the new tools you’ve built? Let's find out.

–

--

#podcast #softwaredevelopment #datascience #apachekafka #streamprocessing

[JavierBlancoCordero]

great questions as always! great answers too! would love to watch a second part of you two!

[MichaelRosam-nozt]

Python as a low barrier, high ceiling stream processing tool for the masses, Gleam as a stream processing tool for Kris!

[lukashei]

Very interesting talk, thanks

Btw, not sure you're already aware but the Spotify version of this episode has somewhat broken sound.

[VolodymyrMoon]

A very long comment, which doesn't make any sense, well at least for you, my dear reader. But! This comment helps these "clever" youtube algorithms to to rank this video better. p.s. Thank you Kris and Tomas.