How to collect logs with Fluentd

In this episode, we will try to explain how to use Fluentd in a K8S cluster
This episode covers the following topics :
- Presentation of Fluentd
- How to parse a file
- How to deploy Fluentd in K8s
- Tutorial

Links Shared during this episode :

Links:

Timestamps
00:00 - Introduction
00:36 - Welcome
01:37 - Presentation of Fluentd
07:51 - How to parse a file
11:05 - How to deploy Fluentd in k8s
15:49 - Tutorial - Configuration of Dynatrace
22:25 - Tutorial - Creating a custom Fluentd Docker container
26:07 - Tutorial - Deploy Fluentd to forward log streams to Dynatrace
34:29 - Tutorial - How to visualize logs in Dynatrace
37:37 - Tutorial - How to build a pipeline using Fluentbit and Fluentd
43:38 - Conclusion

[ctazyfrick]

Hello awesome video, I used your guide to deploy it in a 3 node cluster and its working perfectly. But I have a log file in a nfs directory accessed by all the nodes and now there are 4 duplicates of these logs in flunetd. If you have any idea how to fix that I would appreciate it. Thank you again for the tutorial!!

[jig]

Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again

*edit when I say labels I’m referring to labels used in Loki and visible in Grafana

[shokoofahosseini]

Thanks for your learning, but the back ground music was annoying

[AjayKumar-ijpj]

Hi, I need to centralize the log's into the open search dashboard by using only fluentbit or fluentd in windows OS for my project and should not use K8S. Can anyone help me out here.

[tlikesh]

Hi. can you help me with how to insert the certificates path in my td-agent.conf file.
I have to pass Syslog to open search so I am using the Syslog input plugin and OpenSearch output plugin.
I have to pass a certificate to prevent SSL errors can you please help me with how to do it?

[marcellasilvasanches]

Hello, can you tell me if I can upload a Docker image of a FluentD and make an Http endpoint available for application logging? Please