Elasticsearch anti-patterns and bad practices to be aware of

I've seen and done some things in my time. Some of those things were glorious. Some turned out to be horrible, horrible mistakes. In this video, I speak about some of those things that involved Elasticsearch.

Wonderful as it is, Elasticsearch won't solve all your problems, and may even cause you some sleepless nights if used inappropriately. You might not even need it. I'd like to help you avoid the mistakes I've made and seen while building and managing clusters. It feels good to get this off my chest.

[meitsionmina]

I just have say that you are a great presenter! I've only done minor stuff with ElasticSearch but there were lots of things to consider if our company decides to expand the use cases that are implemented with help of ES.

[samb]

Speaking as somebody who is setting up their first proper Elasticsearch cluster at the moment, this stuff is gold. So often you don't hear about the practical realities of new tech, people only mention the shiny bits, and it's great to go into this project with my eyes open.

[kickinit]

Great tips. I ran a in-house ELK cluster for 6 years and you're absolutely right. The skill needed to setup, maintain, optimize and grow an ELK cluster in many ways does not mirror RDB at all.
Prepare for some late nights unclogging bottlenecks, upgrading and optimizing (and many times rewriting) ingest pipelines. Not to mention the network costs retrieving data from cloud providers and dealing with out-of-date plugins. Lot's to think about. Cheers.

[jamesa]

Certainly can attest to point #1 - We went into our project (hundreds of millions of records, for full document text search) thinking we would have completely flattened data.

Later on in the project we saw a serious need for some form of hierarchy/relationship - I see now why ES HIGHLY recommends flattened data approaches - the hasParent/hasChild queries can be complicated and super expensive, especially for aggregations.

[deez_narts]

I can attest to the challenges you described. I’m the only one in my organization using elasticsearch. Everyone else is afraid to touch it. I do not blame them.

[ankurleonardo]

You deserve at least 10 million subscribers. All watching with zeal.

[atom_]

These are great points - within an organisation where elastic ends up in production without really fleshing out the way it should be used and having teams onboarding their data without true understanding of the data modelling that needs to be done and understood upfront is a major pain to fix later down the line. Common fields are essential (i would say critical) to make most of it when trying to get your data out in a useful way.

[abdirahmann]

am currently building a project that requires semantic search, i wanted to pay someone soo quickly to get this done (i can do it myself but i don't want to spend the time), i thought ELK would be such a breeze, thanks for saving me the time and money :)

[toughcoding]

The worst thing is when you meet Solution Architect who want you to increase index mapping total_fields limit just to fit badly design mapping. Good point!

Good sound quality. wish to have same one day

[guyholmes]

Hi @GeorgeBridgemanData - can Elsatic search be used for types of files that it may never have seen before and "shown" how to parse them?

[sidalisaadi]

I was planning to run a elastic search image for work on docker, thinking it would be a simple process. this video is an eye opener, now I'm not sure if I should go for it or not, our use case is something like yelp with much less data

[nathansherburne]

How about using Elasticsearch for only term searches and doc id queries? Bad use case?

[eladitzkovitch]

Hi George, how can I contact you in person? We would like to invite you as a guest to a function of ours to talk about Elasticsearch.

[mksmurff]

Well that's just put me off completely