From Batch to Streams: Building Value from Data In-Motion

Ricardo Ferreira discusses the risks of designing siloed-based systems and how streaming data architectures can become a solution to address competitiveness.

#SoftwareArchitecture #EventDrivenArchitecture #Streaming

[francksgenlecroyant]

What an amazing presentation Ricardo!!

[willdmindmind]

The difference between a commit log (rdbm) and a stream store is the fact that with streaming you don't necessarily have a fixed schema in advance. In a RDMS the schema is defined up front and therefore the basis of the optimizing the performance of the system. Streaming works fine in a ad hoc big data environment where you are putting the work of figuring out what is in the data into something like Hadoop or other big data search/analysis engines using aggregation functions or some other method. The problem is that this is not necessarily a system that allows for real time ad hoc querying or reports and requires a data process pipeline to extract any real insight outside the fixed transactional behaviors build in advance. RDBMs allow for ad hoc queries that can be used to do the same thing without the overhead of hadoop and other big data oriented processes.