In Memory Data Streams with Jet (Neil Stevenson, Solutions Architect at Hazelcast)

Java 8 gave us a new language abstraction, the stream, which is good but not good enough.
We have some nice abstractions that enable us to code succinctly, and the JVM can exploit multi-processor machines for internal parallelisation. But fundamentally, the implementation is not distributed, we can’t pass data from JVM to JVM cleanly for external parallelisation.

In this talk we’ll look at Jet, a new open-source offering from Hazelcast, that solves this very problem.

The core construct is a DAG – directed acyclic graph – that runs across multiple machines and allows you to plumb together a pipeline of data from place to place, so a thread in one JVM can emit Java objects that a thread in another JVM takes as an input stream. All this happens in-memory, so it’s super fast and you can scale across as many JVMs as you need to boost the parallelisation.

For the developer, familiar constructs such as maps and filters are available so it’s easy to get going, and there’s a library of connections being developed so you can plug in existing data sources and data sinks with minimal effort.

[tsabxyooj]

Dear Neil Stevenson, Can hazelcast compute the Video and Image stream generated from IoT devices?