AWS Elasticsearch Tutorial : What is Elasticsearch and How to Create it and Use it

In this AWS ElasticSearch tutorial you will learn all what you need about Amazon Elasticsearch, how to use elasticsearch, basic elasticsearch commands and more.

What is Elasticsearch

Elasticsearch is a free open-source analytic and search engine which can be used for searching our own logs, monitoring them and more.

What is AWS Elasticsearch / AWS ES

AWS Elasticsearch is a service which makes it easy to deploy and use Elasticsearch in the cloud, by using it in AWS it will be easier to cluster and scale, it can be used also for monitoring your nodes and have multiple configuration.

After creating your AWS ES you will have the options to uploaded your data files to be analyzed and indexed by AWS ES.

The AWS ES upload can be done using:

Index or Bulk API using coding options to accomplish this. (index details , bulk details)
Loading bulk using logstash. (For details check here)
Stream CloudWatch logs to AWS Elasticsearch. (Documentation)
Use third party plugins, which some will give you options to bulk copy, streaming or load from AWS S3. (Ex.: Amazon S3 River plugin for ES)
Using CURL
And everyday more solutions are coming......
How to create AWS Elasticsearch

First you need to create a cluster in the AWS Cloud and that will be your Elasticsearch domain, as any of the other AWS similar services you will be able to choose the number of instances in this domain and their hardware type, for creating your Elasticsearch you can use AWS CLI, AWS SDK or the ES console.

What are the steps of creating your first AWS Elasticsearch?

You create the domain from AWS ES console.
Configure and attach the policy you will use.
Configure and/or upload the files/data will be analyzed by AWS Elasticsearch
Now you can use your AWS ES domain for searching and analyzing the uploaded data

Above video tutorial showing how to create and use AWS Elasticsearch as a quick start

Elasticsearch is an open-source search engine which can be installed manually on your node or you can create a domain on the cloud by using AWS Elasticsearch, below are basic Elasticsearch commands which you can run from any Linux server which has an open access to the Elasticsearch node:

#1 Check your Elasticsearch cluster health: (replace below bold/colored Elasticsearch node with your AWS node or localhost if you are running it from your direct installed instance)

_cat/health?v -- This parameter which used to inquiry the health of the node.

Sample output:

1492895570 14:12:50 513643930529:dotsways green 2 2 42 21 0 0 0 0 -- 100.0%
#2 List AWS Elasticsearch nodes

_cat/nodes?v -- Is used to list the Elasticsearch nodes

curl 'replace-with-your-node-name/_cat/nodes?v'
Sample output:

x.x.x.x 14 98 2 0.03 0.10 0.13 mdi -- bcqocrw
x.x.x.x 14 97 2 0.06 0.11 0.11 mdi * WDBCOid


#3 Create new index

/usersz/external/1 -- Usersz is the name of the new index we want to create.
?pretty -- Is used to have a clean output in separate likes.

Then i am passing a JSON format which has a name of dotsway and a type of website.

curl -XPUT 'replace-with-your-node-name/usersz/external/1?pretty' -d
'{
"name": "Dotsway",
"type":"website"
}'
Sample output:

{
"_index" : "usersz",
"_type" : "external",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"created" : true
}


#4 List indices

/_cat/indices?v -- Is used to list the Elasticsearch indices

curl 'replace-with-your-node-name/_cat/indices?v'
Sample output:

green open example exXUF8zyR1WLMEAY5Lw5SA 5 1 0 0 1.2kb 650b
green open usersz -xrwTiJMQGSRsOJY98V8Hg 5 1 1 0 7.7kb 3.8kb
green open .kibana _dUmh9oJTA2comS1Jjxg-g 1 1 1 0 6.3kb 3.1kb
green open customer yHPoC0-rRaKeHtHKi18OQQ 5 1 1 0 7.7kb 3.8kb
green open example2 Nh62hsm3TwC3qWDMZJHlvw 5 1 0 0 1.2kb 650b


#5 Retrieving:

curl -XGET 'replace-with-your-node-name/customer/external/1?pretty'
7. Delete an index:

curl -XDELETE 'replace-with-your-node-name/customer?pretty'
Those are the basic AWS Elasticsearch commands which can get you started and test the service.

[skdogra]

Simple yet powerful demonstration.. thank you

[JP-vgvl]

i got lost on 6:39 . what server did you connected? is it the Elasticsearch or did you create an EC2 instance before?

[Hackenbaker]

Excelent first approach for ES. Thanks a lot!!!

[networkfreddy]

Video is 2x better if you watch it sped up :)

[LazyProductionzzzzz]

Why is it not recomended to set the acces policy in production to : allow open acces to domain ? I am building a web application and want all users to be able to acces the domain name and be bale to use the elastic search service, so it would be appopiate in my use case, right ?

[haneulkim]

how did u connect aws es to ur command line?

[saivarunbachu]

Does AWS ES only allows JSON data if not how to load XML data please help?

[avimehenwal]

I dont understand, the index was created for What? What are the contents of search index? Did I miss you specifying what to index in video?

[carlotta]

my cluster health status is 'yellow' how can I fix this? I also noticed that you were able to create 2 nodes, and multiple shards while following your steps, my cluster had only 1 node and 1 shard created.

[sompalliharish]

Any alternate options for AWS elasticsearch service

[ariben-elazar]

Creating an index in 6.0+ is now:

curl -XPUT \
-H 'Content-Type: application/json' \
-d '{ "name": <name>", "type": "website" }'

[HassenTaidirt]

Good intro but why have you been half way? Will be a complete intro if at least you show how to search for a specific item (this is the main use case of Elasticsearch)

[vyangbaan]

what is about implementation with system?

[diogosilva]

How can I use alerts with amazon elasticsearch alert?

[ivandres]

Nice video, but we appreciate the ABSENCE of loud intro music.

[sergeileshchinsky]

Not a word about actual use case. Wasted time.

[Anomaelea]

You can't hide that Iranian accent mate

[tommybenshaul]

Kid, Who are you? what experience do you have?