SparkUCX – RDMA acceleration plugin for Spark

Peter Rudenko, Mellanox Technologies
Peter Rudenko is a software engineer in Mellanox High Performance Computing team, focusing on accelerating data intensive applications, developing UCX communication library and various big data solutions.

Many HPC clusters are shared with Big Data crunching frameworks for better resource utilization. It makes a lot of sense to use that RDMA enabled network to accelerate the Big Data framework applications, such as Spark or Hadoop. In this session I would like to present the Java binding developer in the UCX open source project. We will review the UCX architecture and see how this new Java UCX API’s allows fast and simple Java application integration into the RDMA subsystem. I will present the SparkUCX shuffle plugin we created, based on the jUCX API’s and show the performance improvement it provides.