Interacting with Elasticsearch via Python - A Primer in 2021

Learn how to interact directly with Elasticsearch using Python as well as found out some use cases for doing so. This video explores using Python against Elasticsearch based on a project to implement custom lifecycle management strategies for index deletion, rollovers, allocation, and cloud accounting. H & A Security Solutions is sharing some of our experiences, both good and bad, with Elasticsearch and how Python can lift barriers you come up against.

Link to Elastic ILM project used during the video can be found here:


Do not forget to subscribe to this channel for updates on future videos. Also, H & A Security Solutions provides aid to clients on a daily basis to use, implement, and maintain Elastic stacks as a SIEM.


Speaker: Justin Henderson, CEO H & A Security Solutions LLC

Justin is the co-founder of H&A Security Solutions, LLC, a company that deploys, maintains, and tunes SIEM, NSM, and other solutions for organizations. Justin also maintains one of the largest security onion deployments in the world with over 1200 network sensors. He is a passionate security architect and researcher whose experience in cybersecurity started at the age of thirteen when he began providing professional services to organizations. Justin was the 13th GSE to become both a red and blue SANS Cyber Guardian and holds over 60 industry certifications. As the author of SEC555 and co-author of SEC455 and SEC530, he’s able to bring his encyclopedia of IT knowledge into the classroom.

[tobypass]

Love this video. I was able to follow along. Rather than just using your repo, I downloaded it with git then coped the relevant functions and log files from es.py into my script so I could see how it worked. Really good walkthrough - appreciate the knowledge sharing. Thank you.

[adliwahid]

Thanks for this! Been putting off ILM for too long - the video with code really helps demystify certain things - cheers!

[kevinz]

Fantastic video with extremely clear explanation. Fantastic teacher thank you

[IvarsRuza]

wow U saved my headache - enrichment after the index is set. I was doing it in logstash - but thats a lot of planing - I do always forget smtg and had rebuild the index from syslog again and again.

[TheMiguel]

Can I import 2 schemas from elasticsearch to dataframes and perform joins on the jupyter notebook?

[mp]

Thank you. How could I access the elastic search records for a given period of time for a given index?

[lauragalera]

pretty useful. However, is there a way that, when you transform the response to a dataframe, to add a column _score? Seems the response does not include it tho...

[riyanirawan]

hello, I want to ask. I have executed the query and in the index data I entered. but the csv/json file is not uploaded to the file. I checked the index pattern, the health status is yellow and the file size is 208b???

[pulkitdikshit]

can we create index without using elastic search python module? by using endpoint url and how to bulk upload data on those created index?

[pismed]

can you help me i will connect to elastic cloud with file python