Building Complex Data Analytics Pipelines with Ray - Qingqing Mao, Dascena

Building Complex Data Analytics Pipelines with Ray - Qingqing Mao, Dascena

"Building scalable data analytics pipelines is challenging, especially when different subtasks may have different computational requirements and interdependencies. It becomes more challenging when you need to serve enterprise customers who have strict data security and privacy policies and require on-premise deployment. The scaling requirement and computational capacity often vary widely from site to site.

We have been using Ray to build natural language processing pipelines and healthcare analysis pipelines. The highly efficient serialization using a shared-memory object store is a perfect fit for handling our data-intensive jobs. Ray helps us narrow the gap between data science and engineering, and it enables our data scientists to write high-performance and cost-efficient data analytics pipelines that can scale. "