Massively Parallel Hyperparameter Tuning

Показать описание

Ameet Talwalkar, Carnegie Mellon University Assistant Professor of Machine Learning, Chief Scientist at Determined AI, and leading expert in the area of AutoML. Ameet focuses on the problem of massively parallel hyperparameter optimization.

Modern learning models are characterized by large hyperparameter spaces. In order to adequately explore these large spaces, we must evaluate a large number of configurations, typically orders of magnitude more configurations than available parallel workers. Given the growing costs of model training, we would ideally like to perform this search in roughly the same wall-clock time needed to train a single model. We tackle this challenge by introducing ASHA, a simple and robust hyperparameter tuning algorithm with solid theoretical underpinnings that exploits parallelism and aggressive early-stopping. Our extensive empirical results show that ASHA outperforms state-of-the-art hyperparameter tuning methods; scales linearly with the number of workers in distributed settings; converges to a high quality configuration in half the time taken by Vizier (Google's internal hyperparameter tuning service) in an experiment with 500 workers; and competes favorable with specialized neural architecture search methods on standard benchmarks.

Ameet's Bio: Ameet Talwalkar is an assistant professor in the Machine Learning Department at Carnegie Mellon University, and co-founder and Chief Scientist at Determined AI. His primary interests are in the field of statistical machine learning, including problems at the intersection of systems and learning. His current work is motivated by the goal of democratizing machine learning, with a focus on topics related to the scalability, automation, and interpretability of learning algorithms and systems. He led the initial development of the MLlib project in Apache Spark, is a co-author of the graduate-level textbook 'Foundations of Machine Learning' (2012, MIT Press), and created an award-winning edX MOOC about distributed machine learning.