filmov
tv
Real Time Stream Processing with Python at Scale (Alejandro Saucedo) (UK) - PyCon HK 2020 Spring
Показать описание
This talk will provide a practical insight on how to build scalable data streaming machine learning pipelines to process large datasets in real time using Python Asyncio, Kafka, Faust, SpaCy and Seldon.
We will be covering a case study performing automated content moderation on Reddit comments in real time. Our dataset will consist of 200k reddit comments from /r/science, 50,000 of which have been removed by moderators. We will be handling the stream data in a Kafka cluster, and the stream processing will be handled using the stream processing library Faust. We will be running the end-to-end pipeline in Kubernetes with various components legeraging SKLearn, SpaCy and Seldon.
We will then dive into fundamental concepts on stream processing such as windows, watermarking and checkponting, and we will show how to use each of these frameworks to build complex data streaming pipelines that can perform real time processing at scale.
Finally we will show best practices when using these frameworks, as well as a high level overview of tools that can be used for monitoring, including Grafana and Kafka Manager.
Level: Intermediate
Speaker Bio: Alejandro Saucedo
Alejandro is the Chief Scientist at the Institute for Ethical AI & Machine Learning, where he leads the development of industry standards on machine learning bias, adversarial attacks and differential privacy. Alejandro is also the Director of Machine Learning Engineering at Seldon Technologies, where he leads large scale projects implementing open source and enterprise infrastructure for Machine Learning Orchestration and Explainability. With over 10 years of software development experience, Alejandro has held technical leadership positions across hyper-growth scale-ups and has delivered multi-national projects with top tier investment banks, magic circle law-firms and global insurance companies. He has a strong track record building cross-functional departments of software engineers from scratch, and leading the delivery of large-scale machine learning systems across the financial, insurance, legal, transport, manufacturing and construction sectors (in Europe, US and Latin America).
We will be covering a case study performing automated content moderation on Reddit comments in real time. Our dataset will consist of 200k reddit comments from /r/science, 50,000 of which have been removed by moderators. We will be handling the stream data in a Kafka cluster, and the stream processing will be handled using the stream processing library Faust. We will be running the end-to-end pipeline in Kubernetes with various components legeraging SKLearn, SpaCy and Seldon.
We will then dive into fundamental concepts on stream processing such as windows, watermarking and checkponting, and we will show how to use each of these frameworks to build complex data streaming pipelines that can perform real time processing at scale.
Finally we will show best practices when using these frameworks, as well as a high level overview of tools that can be used for monitoring, including Grafana and Kafka Manager.
Level: Intermediate
Speaker Bio: Alejandro Saucedo
Alejandro is the Chief Scientist at the Institute for Ethical AI & Machine Learning, where he leads the development of industry standards on machine learning bias, adversarial attacks and differential privacy. Alejandro is also the Director of Machine Learning Engineering at Seldon Technologies, where he leads large scale projects implementing open source and enterprise infrastructure for Machine Learning Orchestration and Explainability. With over 10 years of software development experience, Alejandro has held technical leadership positions across hyper-growth scale-ups and has delivered multi-national projects with top tier investment banks, magic circle law-firms and global insurance companies. He has a strong track record building cross-functional departments of software engineers from scratch, and leading the delivery of large-scale machine learning systems across the financial, insurance, legal, transport, manufacturing and construction sectors (in Europe, US and Latin America).
0:06:20
0:13:23
0:34:08
0:28:11
0:35:32
0:00:49
0:27:37
0:07:50
0:00:59
0:46:52
0:26:11
0:17:16
1:06:34
0:09:43
0:48:58
0:50:57
0:49:19
0:11:24
1:05:00
0:04:03
0:21:58
0:31:30
1:27:48
0:27:25