ML-Based SQL Query Resource Usage Prediction

Alluxio Day XV
September 15, 2022

Speakers:
Chunxu Tang (Senior Software Engineer, Twitter)
Beinan Wang (Senior Software Engineer, Alluxio & PrestoDB Committer)

With the advent of the Big Data era, it is usually computationally expensive to calculate the resource usages of a SQL query. Can we estimate the resource usages of SQL queries more efficiently without any computation in a SQL engine kernel? In this session, Chunxu and Beinan would like to introduce how Twitter’s data platform leverages a machine learning-based approach in Presto and BigQuery to estimate query utilization with 90%+ accuracy.