Fluentd on Kubernetes: Log collection explained

In this video we take a look at log collection on a kubernetes cluster.
We take a look at the fluentd dockerfile, how to run it on each kubernetes node using a daemonset.
More importantly is the configuration required to tell fluentd how to collect logs. We also look into where logs are stored on each kubernetes node.
Finally we see how to ship the collected logs out of the cluster.

Checkout my Introduction to Logging video:

Checkout my Introduction to Fluentd video:

Checkout the source code below 👇🏽 and follow along 🤓

Also if you want to support the channel further, become a member 😎

Checkout "That DevOps Community" too

Source Code 🧐
--------------------------------------------------------------

If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)

Kubernetes Guide for Beginners:
---------------------------------------------------

Kubernetes Monitoring Guide:
-----------------------------------------------

Kubernetes Secret Management Guide:
--------------------------------------------------------------

Like and Subscribe for more :)

Follow me on socials!

Music:

[amukher]

This is perhaps the best single resource to understand how to setup fluentd (and by extension Fluent Bit) for a k8s environment. There's so much of crud out there in the form of blog articles and documentation, not to mention a book by a certain well-known publisher, all of which are completely useless. It really should be a lot simpler - this video takes care of it.

[videomaster]

Bro, trust me you are "that guy"!
every minute is valuable and go straight to the point!!
Big thanks!!

[Jak]

I've got to say, I really appreciate these videos. As an intern trying to get into the field of DevOps, your tutorials are always my go to. Fast, concise and extremely helpful. Thanks alot!

[sravankumar..]

You are the one stop to clear all my questions on kuberentes. Just I need to watch all your kubernetes videos to clear all my questions, great man 🙏🏽🙏🏽 You are a wonderful guy, love from India.

[bisuaws]

You are the genuine Devops Builder, many thanks 😊

[videobrice]

Thanks for this excellent and exhaustive tutorial !

[devops-techstack]

Thank you so much for sharing this knowledge . It helped a lot while i was struggling to send logs to Stackdriver.

[Misanthrope]

Another brilliant video mate 🤟 please keep them coming!

[arpanchakraborty]

Amazing, thanks so much, very simply explained yet very informative and almost all you need to know to start off with EFK.

[vijaytamhankar]

Nicely explained. Easy to follow steps 👍

[ThiagoSilva-]

Thanks for the teaching.
You are amazing !!!

[irfannian]

For a weird reason, I am unable to find file-test.log in my /tmp/ folder as shown in 20:45. Prior to that, everything seems to be executing fine.

[oraculotube]

Thank you very much to share your knowledge. much appreciate that.

[rabithbo]

Thanks so much, it is amazing tutorial.

[RahulKumar-kbj]

Is it possible to run it as sidecar container ?

[ckalas]

new video idea: the implications of k8s dropping docker as default container runtime

[rahulsingh-iqgd]

Sheer knowledge… only, thank you so much

[milad]

hi, how can store logs per namespace in different index?

[amukher]

A basic question. How does Fluent Bit enrich the logs with kubernetes container, pod, and namespace information? Does it derive it from file naming conventions (and then from kube-api as you mentioned)?

[bhuvanparashar]

Hi, what will happen if I use more than one source targeting the same path?