Real-time data analysis using ELK – Jettro Coenradie [Luminis DevCon 2015]

You are a developer, create applications that generate logs. You would like to monitor those logs to check what the application is doing in production. Or you are an operator in need for information about the whole platform. You need logs from the load balancer, proxy, database and the application. If possible you would like to correlate these logs as well. Maybe you are an analyst and you would like to create some graphs of the data you obtained. If one of these roles is you, the chance is big you heard about ELK. This is short for Elasticsearch, Logstash and Kibana. The goal for these projects is to obtain data (logstash), store it in a central repository (elasticsearch) to make it searchable and available for analysis. Having all this data is nice, but making it visible is even better, that is where Kibana comes in. With Kibana you can create nice dashboard giving insight into your data. ELK is a proven technology stack to handle your logs. During this talk I will present you the complete stack. I’ll show you how to import data with logstash, explain what happens in elasticsearch and create a dashboard using Kibana. I will also discuss some choices you have to make while storing the data, go into a number of possible architectures for the ELK stack. At the end you have a good idea about what ELK can do for you.

Jettro Coenradie
Software Enigeneer & Fellow Luminis
Jettro Coenradie is a Fellow at Luminis Amsterdam. He has been delivering applications for over 15 years. In these years Jettro has learned the importance of a close connection between development and operations. Therefore he is a strong believer in the devops movement. One type of handover from development to operations is documentation, but the best one is good logs. Jettro likes to try out new stuff. That is why he recently got his motorcycle drivers license. On a motor bike you tend to explore different routes to get the best our of your bike and have fun while doing the thing you need to do, get from A to B. When exploring new technologies he also likes to explore new routes to find better and more interesting ways to accomplish his goal. Jettro rides an all terrain bike, he does not like riding the same ground over and over again. The same is valid for his technical interest, he knows about backend (elasticsearch, mongodb, spring data, spring integration) as well as front-end (AngularJS, Sass, Less) and mobile development (iOS, Sencha touch).