Efficient BERT Full Talk: Find your Optimal Model with Multimetric Bayesian Optimization

In this webinar, we explore how to reduce the size of BERT while retaining its capacity in the context of Question Answering tasks. We pair distillation with Multimetric Bayesian Optimization. By concurrently tuning metrics like model accuracy and number of model parameters, we will be able to distill BERT and assess the trade-offs between model size and performance. This experiment is designed to address two questions through this process:

🔵 By combining distillation and Multimetric Bayesian Optimization, can we better understand the effects of compression and architecture decisions on model performance? Do these architectural decisions (including model size) or distillation properties dominate the trade-offs?
🔵 Can we leverage these trade-offs to find models that lend themselves well to application specific systems (ex: productionalization, edge computing, etc)?