Build Semantic-Search with Elastic search and BERT vector embeddings. ( From scratch )

Welcome to my comprehensive coding tutorial where we'll discuss the process of creating a powerful semantic search engine from scratch! In this video, we'll combine Elasticsearch, SBERT machine learning models, and Streamlit to build a robust search application that understands the meaning behind our queries.

What we'll discuss:

Setting Up Elasticsearch: We'll start by setting up Elasticsearch, a highly scalable and flexible search engine, as the backbone of our semantic search system. We'll learn how to index your data effectively for efficient searching.

SBERT Embeddings: Dive into the world of SBERT (Sentence-BERT) and discover how to use pre-trained language models to transform text into meaningful numerical representations that capture semantic information.

Semantic Search Algorithms: Learn how to implement semantic search algorithms that can find contextually relevant results, going beyond traditional keyword-based searches.

Streamlit User Interface: We'll create an interactive and user-friendly front-end using Streamlit, allowing you to search and explore your data effortlessly.

By the end of this tutorial, you'll have the skills to build your own semantic search engine and customize it to suit your specific needs. Whether you're a developer looking to enhance search capabilities or a data enthusiast interested in understanding semantic search, this video is your gateway to creating intelligent and context-aware search solutions.

Don't miss out on this in-depth coding tutorial! Join us as we unlock the potential of semantic search using Elasticsearch, SBERT, and Streamlit. Subscribe, like, and share to stay updated with the latest tutorials on building advanced AI-powered applications. Let's start searching smarter today!

0:00 - Intro
0:28 - How end product will look like!
1:07 - Architecture
3:37 - Setup Elastic Search server
4:43 - Connecting to Elastic Search using Python API
9:14 - Data Prepration
12:21 - Vector Conversion using S BERT model
16:50 - Creating index in Elastic Search for KNN search
23:27 - Data ingestion in Elastic Search index.
26:58 - Writing search function to retrieve the data from Elastic Search!
30:57 - Streamlit UI
35:09 - Yayy!! Using Final Search UI.
38:25 - Thank You

LEARN PYTHON STEP BY STEP :

YOU CAN FIND ME HERE AS WELL :

[dkobia]

Nicely done. Thanks for the walkthrough

[sulaiman]

I think you should consistently upload videos .
More video like this very helpful ✅
❤❤❤🙌🏻

[margipatel]

Thank you for sharing! Very clear, thorough, well-paced and learner centered. What an amazing educator!

[judepops]

Incredible video, you are amazing! Thank you for the concise outline of the video and explanations of all the key concepts. You have a new sub :)

[bigm]

Wow! This is amazing work you did here. Your work is very clear, the process well well explained 👏. I was looking into setting up vector embeding search with Elastic and you've clarified the whole process and fundamental understanding of such a solution.

[mehulmak]

How to run this as different services for es and streamlit via docker compose? Also if I need to use any other vectordb is there any way to do it?

[arpitsingh]

great tutorial, wanted to ask if elastic search is the best option for basic l2 or cosine vector similarity insted of some online or offline vector db based methods as it uses approximate nearest neighbours ( ANN ), also this approach cannot be used in cases where false positives are not acceptable such as cache implementation or medical data semantics, can you suggest something that resolves these problem statements 😢

[snehancoghosh]

Hey man great video. it would be amazing if you could teach us how to do the same thing using Opensearch and docker

[souvikdas]

I have watched the video, need advice from you on a similar use case. So I have API log data and for that I want to create a similar search application. Is it possible for the search application to answer questions like the ones below? -
"List all failed transactions between Merchant_B and any bank.",
"Find transactions between Merchant_A and Bank_X." etc.

[thelocalguide]

Hello abid Is it possible to convert a specific field in elasticsearch that already contains data into vector and then use semantic search in it. Great Content

[afederici]

This was a great video! Thank you so much. I enjoyed the format (very much to the point, with a nice clarification at the beginning) and you presented it really well. I would love to see a video in the future that maybe explains how picking a custom model could help being charged a lot by ES's ML nodes. In other words, how do you think we could use custom models to be able to use smaller ES ML nodes (e.g. 1GB or 2GB, vs the 4GB ESRE) and be charged less? Thanks again

[satyashah]

True, 'shards_acknowledged': False, 'index': 'series120'}) I am getting this error
How can I solve this?

[eduardovarela]

At final part of code i have this error BadRequestError(400, 'illegal_argument_exception', 'Invalid type: expecting [_doc] but got [_knn_search]')

[rajeevmishra]

hi @abid, thanks for the video, can you help me in combining multiple column data for vector search instead of just one as somtimes the description does not contain everything and the customer may look for things from other column. i hope, you got it. please help.

[valentind.]

Thank you for sharing, it really was a pleasure to follow that tutorial. I just had a video recommendation about the latest improvements of ElasticSearch and now I am wondering if that video is still relevant as of today considering the improvements that have been made (ELSER) ?

[seriesophile]

you didnot provide the repository link to work with, or the dataset path . please share the repository in git. it would be quite helpful coz i am currently working on it

[AustinMark]

great video. I was also wondering what vs code plugin you used that color coded your python indentations?

[dmvrlnb]

great work amigo
i just had one question please :
what should i do in order to search by all the fileds (price, brand, gender ...ect ) ???

[clouds]

Any chance you would show us how to implement this but using cosine similarity? :), Amazing video, helped me learn a lot!!

[leamon]

Hi, thanks for the tutorial. Could you also make one to compare elasticSearch with other vector database solution in terms of semantic/vector search?