[Scala Central] Courtney Robinson - Reactive is Dataflow, with Akka Streams

Courtney Robinson, Software Engineer, FortyTwo Data.

We've been experimenting with Akka Streams and adopting a fully reactive model in a way that gives us the following properties: Scalability; Reactive; Elastic; Self healing.

By being reactive we get certain system properties for free including the typical fast producer, slow consumer situation. We are defining interfaces to interact with "resource providers" e.g. Mesos/Marathon, DC/OS or Kubernetes to enable the system to automatically scale up more instances depending on the current load profile and the service's configured SLA. This is all driven by metrics automatically captured. As load reduces and it auto scales back down enough to comfortably meet SLAs.

This talk will be a high level overview of the design of the library, where we're at with it and the direction we're trying to take it to. We're in the early stages of this work but already getting better throughput with less resources than our legacy spark setup and with plans to migrate the entire platform to using this new model.