Stateless Fluentd with Kafka - Steven McDonald, Usabilla

Stateless Fluentd with Kafka - Steven McDonald, Usabilla

Fluentd is typically deployed as a central aggregator to which everything sends its logs for processing and routing. This superficially simple approach was found to be inadequate at Usabilla. Errors in one part of the processing chain often had knock-on effects elsewhere, leading Usabilla's SREs to search for a more failure tolerant design. Steven will introduce the new stateless fluentd deployment at Usabilla, built around Kafka as a centralised, highly available log buffer. He will also introduce the new components that have been developed to adapt fluentd to be completely stateless, as well as how logs are reliably fed into Kafka from hosts all over the world. Finally, there will be a brief overview of the challenges still remaining.

[SeanGlover]

I found this talk really helpful for deciding to pursue Kafka w/out fluent

[DuongNguyen-flqd]

Do you have the Github link for the system that you built, sir? I'm interested in building one for testing purpose, but I'm now get stuck with the problem of the data stream between Fluentd and Kafka.
Very appreciate for your presentation, sir!