Sparse Neural Networks: From Practice to Theory

Atlas Wang
Assistant Professor, Electrical and Computer Engineering
The University of Texas at Austin

Abstract: A sparse neural network (NN) has most of its parameters set to zero and is traditionally considered as the product of NN compression (i.e., pruning). Yet recently, sparsity has exposed itself as an important bridge for modeling the underlying low dimensionality of NNs, for understanding their generalization, optimization dynamics, implicit regularization, expressivity, and robustness. Deep NNs learned with sparsity-aware priors have also demonstrated significantly improved performances through a full stack of applied work on algorithms, systems, and hardware. In this talk, I plan to cover some of our recent progress on the practical, theoretical, and scientific aspects of sparse NNs. I will try scratching the surface of three aspects: (1) practically, why one should love a sparse NN, beyond just a post-training NN compression tool; (2) theoretically, what are some guarantees that one can expect from sparse NNs; and (3) what is future prospect of exploiting sparsity.

[aminuabdulsalami]

Awesome !!! Never ever thought sparsity in NN could be this interesting.

[arthurzhang]

a natural question to ask is : can the random sparse pruning apply to GPT-style large models to achieve small models?

[Bert]

Are the slides available somewhere? Thanks!