Aggregations: Solrcloud/Elasticsearch, Druid or HBase - Dragan Milosevic, Zanox

You need to build a highly scalable system for executing aggregation-queries in real-time on big data. But you do not have several weeks to try each and every available technology that supports such queries, and you are not sure which one to pick. We have taken time to build fully functional prototypes and have learned important lessons that can serve as precious time-saving guidelines while deciding on the architecture of your system.

To have an unbiased comparison, we installed each built prototype on a cluster of machines having exactly the same hardware configuration. We estimated the ingestion performance by measuring the time that each prototype needs in order to make the imported records become available for querying. We executed real-user aggregation-queries to measure the response time while simulating various ingestion loads. By increasing the number of machines that are used to run the built prototypes, we were able to estimate the ability of each technology to scale. Finally as a bonus, we will also share our subjective opinion regarding the ease of use, flexibility, customizability, and available community support for each evaluated technology.