Building a Near Real-time Search Engine and Analytics for logs using Solr

Presented by Rahul Jain, System Analyst (Software Engineer), IVY Comptech Pvt Ltd

Consolidation and Indexing of logs to search them in real time poses an array of challenges when you have hundreds of servers producing terabytes of logs every day. Since the log events mostly have a small size of around 200 bytes to few KBs, makes it more difficult to handle because lesser the size of a log event, more the number of documents to index. In this session, we will discuss the challenges faced by us and solutions developed to overcome them. The list of items that will be covered in the talk are as follows.

Methods to collect logs in real time.
How Lucene was tuned to achieve an indexing rate of 1 GB in 46 seconds
Tips and techniques incorporated/used to manage distributed index generation and search on multiple shards
How choosing a layer based partition strategy helped us to bring down the search response times.
Log analysis and generation of analytics using Solr.
Design and architecture used to build the search platform.

[arunmd]

Congrats dude, , happy that I worked with you once upon a time :)

[rakeshkumarjain]

Rahul jain your presentation will a milestone in your future

[satishjain]

Many 2 congratulation Ur Method of Presentation is very Good I Hope its Support in ur Bright Future