Install Elasticsearch Kibana and Logstash with Docker

#elasticsearch #filebeat #kibana #elasticsearchtutorial #logstash #metricbeat #docker #dockercompose #dockercontainer
In this video, we install Elasticsearch, Kibana, and Logstash using Docker Compose. This is a way to bring up a cluster for quick testing and development.

You can get the files from my GitHub ink:

Watch how to install Elasticsearch and Kibana:

Thank you for watching!

[agilebarsfromtimebarsltd.]

Totally awesome, thank you very much.

[mathas]

Thank you for the video. Really appreciate it. Maybe you can add more hands on in ingesting filebeat (including logstash filtering) and metricbeat to monitor firewall metricbeat in kibana

[jonmarkortiz]

Thanks so much for this very simple and well narrated tutorial. I am curious what your approach would be. I currently have my docker-compose file that has the following services - frontend, backend, mongo, and redis. My frontend and backend are referencing builds that point to Dockerfiles that exist in the roots of each directories. The mongo and redis are not and instead referencing the images along with additional meta info. My question is this - wanting to keep my docker-compose file more readable and not make it too enormous, is there a strategy on how to introduce the services for elasticsearch, kibana, and some number of es nodes - es01, es02 etc? In regard to the docker-compose implementation which elastic gives us, is it possible to create an elasticsearch directory, with a Dockerfile that abstracts out more of the docker-compose implementation? Are there examples out there you know of and maybe some key pages in Docker to reference regarding this? Thanks again for all your help. Btw, I am happy to send you a link to my existing repo containing my yml, if it helps you see more clearly. Thanks again.

[Karan-gkjw]

Hey sir the volume you are talking about around 3:30 can we use kafka as the volume

[dv]

Hi, it seems that the main Docker Compose code was borrowed from the elkninja repository, as described in an Elasticsearch blog post. However, there is a significant drawback to this implementation: the generated certificates lack passwords, and no keystores are configured. The author of the blog post mentioned that this setup is suitable for Proof of Concept (POC) purposes, but not for production environments.

[zgxdznv]

hi
thanks for your video
plz make video about rolling upgrade cluster node elasticsearch
i want to upgrade with rolling upgrade but when i upgrade node 1, i give error:
1.master node disconnected, restarting discovery
2.this node is locked into cluster UUID
help me if you can

[patilavinash]

Hi I want to install ELK on a test/production server can you please me for that

[avalagum]

The SSL thingy makes everything look complicated. Is there any setup with only 1 node for elasticsearch without SSL?

[DanielGonzalez-pvmu]

The error message you're seeing is related to Elasticsearch bootstrap checks that are performed when Elasticsearch detects that it is running in production mode. Specifically, the error:

```
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
```

indicates that the `vm.max_map_count` setting on your host is set too low for Elasticsearch to operate reliably in a production environment. This setting defines the maximum number of memory map areas a process may have. Elasticsearch recommends setting this to at least `262144`.

### Fixing the `vm.max_map_count` Issue

To resolve this issue, you need to increase the `vm.max_map_count` setting on your host system. This setting is applied at the OS level, not within Docker containers, so you must set it on the host that runs your Docker daemon.

#### For Linux Hosts

1. **Temporarily (does not survive reboot):**

You can temporarily set `vm.max_map_count` to the recommended value by running the following command on your host:

```sh
sudo sysctl -w vm.max_map_count=262144
```

2. **Permanently (survives reboot):**

To make the change permanent, so it persists across reboots, add the following line to `/etc/sysctl.conf`:

```
vm.max_map_count=262144
```

Then, apply the changes with:

```sh
sudo sysctl -p
```

#### Verifying the Change

To verify that the setting has been applied, run:

```sh
sysctl vm.max_map_count
```

You should see `vm.max_map_count = 262144` as the output.

### After Adjusting `vm.max_map_count`

Once you've adjusted the `vm.max_map_count` on your host, you should be able to start your Elasticsearch service without encountering the previous bootstrap check failure. If you're using Docker Compose, make sure to restart your services for the changes to take effect:

```sh
docker-compose down
docker-compose up -d
```

### This took me a couple of hours to figure out, but it had me stumped as well. hope it helps. I also increased my total ram on the VM to 16gb of ram, and she's pegging around 85% usage. will most likely end up increasing to 20 gb. but I am also looking at decreasing number of nodes. I only just started. thanks to OP. I was stuck on this ELK stack for a while.

[edinsonguzman]

I run the docker-compose up -d and always fail to start the container elk-es01-1, How to troobleshoot this problem?

[therus]

thanx for video, so nice work
but can u share please that docker-compose file and config file for logstash

[Ethan]

what terminals and packages do i need in VS Code?

[Ethan]

I'm actually trying to replicate your setup on my computer but difference is I'm using localhost.

I'm running into issues currently regarding docker socket. There is a bad gateaway connection that causes kibana container to hang up and Exit because it fails to establish a connection with Elasticsearch container.

On my elastic search container, when I do a curl request to localhost 9200, I dont get a response either.

What is going wrong in my setup? Currently on ELK version 8.11.0 across all components.

[irrukvu]

бля епересетээээээээ а где файлики с

[cdmotek]

hello
I enjoyed watching the YouTube video

I added the settings and files as shown on YouTube and ran it, but the same error as Hardy occurred.
✔ Network elasticity created
✔ Container elkdocker-setup-1 Healthy
✘ Container elkdocker-es01-1 Error
✔ Container elkdocker-kibana-1 Created
✔ Container elkdocker-es02-1 Created
✔ Container elkdocker-es03-1 Created
✔ Container elkdocker-logstash-1 Created
dependency failed to start: container elkdocker-es01-1 exited (78)

I wonder if there is any workaround.

And I'm curious how to enter the url for kibana to appear in the browser.

take care

[zhajikun]

I run your docker-compose file but get this error in Kinaba:
FATAL Error: [config validation of value has length [16] but it must have a minimum length of [32].

[arggomes]

Hi Ali, nice explanation, but i am receiving the following error below.

Creating agomes_setup_1 ... done
Creating agomes_es01_1 ... done
Creating agomes_kibana_1 ... done
Creating agomes_es02_1 ... done
Creating agomes_es03_1 ... done
Creating agomes_logstash_1 ... error

ERROR: for agomes_logstash_1 Cannot start service logstash: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/agomes/logstash.conf" to rootfs at mount (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for logstash Cannot start service logstash: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/agomes/logstash.conf" to rootfs at mount (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.