Filebeat + Elk Stack Tutorial With Kubernetes

[awaitingforsunrise]

Michael, thank you so much!
You showed us very useful architecural approach: "setup once, run everywhere". With this approach we don't need to be bothering with logging in our microservices at all, because ELK ingests the console output stictly to Elastic DB.
It took me some time to get this runned with few corrections of your original files (i.e. "ssl_certificate_verification => false" was required to get logstash being able to connect Elastic and now Elastic requires https connection) + it was needed to bootstrap Elastic's password. Besides it, I was able to get this working and adsorb the logs of all of my microservices in ALL namespaces, despite on ELK lives in its own namespace.
Keep it up, nice work!

[josesantos]

You are awesome. Everything worked for me and now I have a complete ELK cluster. Thanks. :-)

[remram]

You seem to be a bit confused about how one deploys a Helm chart. The values.yaml is not "what allows us to override the templates", those are the default values to be overridden using the -f option of helm install. You don't clone the chart to install it

[shaishmuel]

Great video, you helped me a lot! It was very clear and straight forward, hope that you will upload more useful content👏

[nishanthakumara]

Great explanation. please upload more videos related to the Kubernetes 🙂

[HenriqueWeiand]

Great video! Thank you

I wonder if you could review the repo and update the version of the services. It seems not to be working if I try to update the service version

[MidorimaShintaro-om]

Nice Tutorial, It's helping a lot ! Just to ask, When deploying elastic search using helm chart it automatically creates pv, pvc and storageclass but when actually deployed it only created pvc the pv and storageclass is missing .. which eventually resulted into error "pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.." how to resolve this issue ..

[nnmbnmbnmnm]

Thanks for sharing, very good explanation. would you please also share what is the log retention policy? like for how many days it will preserve the logs and then purge them?

[nitinvij]

nice and precise tutorial, wanted to know some changes we might have to do when the container runtime is "containerd" like in my case i am using k3s

[oliverabrahamhamburg]

Finally I made it to connect to the kibana UI, but only using the forward button in Lens.
I still cannot access the UI using my regular kubernetes IP. And I don't understand how you made it to configure the nginx.
I would appreciate if you would spend more time for the next tutorial, because it was way to fast for me.

[usuariousuario]

Thank you very much, you help me a lot. How should work to have the logs of the others containers? I just seen the ELK Stack, but a application with logs is not showing

[VinayBedre]

Learnt a lot with your video. Thanks for sharing.

[ashketchum]

Hi Michael,
firstly thank u for such a concise lecture but there is one problem that I am facing is that elasticsearch is by default configured to take data at https port while we are making logstash send data on http port, so how do I configure elasticsearch to accept data on its http port as because of this no data is being sent to it. Please help with it as I have checked multiple resources but no one has got a solution to it. It would be great if you'd throw some light on it.

[amseshuu]

@Michael Thanks nice video and explanation.. i would like to impersonate roles and permissions in ELK Stack.. like want use the ELK security feature.. to create different logins for dev_user, dashboard_only_user..etc.. segregating app1, app2..with data1, data2.. etc

[nlokesh]

Thanks for the video.
Can you please help me understand..
1. How did you setup external-ip to Kibana Pod?
2. Can I implement this solution in AKS?

I lokk forward to your response..

[Shreddoctor]

Excellent man! Thank you very much for that brother.

[parthibarajanranganathan]

how do i configure my filebeat to get 100's of my pod logs. 100 pod logs -> file beat-> logstash-> elasticsearch. we have nearly 100 pods running in our kubernetes namespace

[po]

you make this look so easy!! great thanks!!

[supermario-dquy]

while installing logstack iam getting error Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "logstash-logstash-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first

[JackReacher]

couldn't able to run kibana like you did, the current chart of kibana has pre and post actions and it is causing all the problem while installing and uninstalling the chart.