Predicting the parameters of a neural network without training it | Trustworthy AI | Webinar

Reducing barriers for deep learning practitioners to develop neural networks is one step towards democratizing deep learning, by making the technology more accessible to smaller players in the field. This AI for Good Discovery will explore the key themes in Graham Taylor’s research around removing barriers for deep learning practitioners who lack the background or resources to work with cutting-edge models that require advanced forms of hardware parallelism. Collaborating with Facebook AI Research (now Meta), his team developed a technique to initialize diverse neural network architectures using a “meta-model”. This research challenges the long-held assumption that gradient-based optimizers are required to train deep neural networks.

Astonishingly, the meta-model can predict parameters for almost any neural network in just one forward pass, achieving ~60% accuracy on the popular CIFAR-10 dataset without any training. Moreover, while the meta-model was training, it did not observe any network close to the ResNet-50 whose ~25 million parameters it predicted. Like the team’s 2020 work to reduce the computational requirements of GANs, this talk highlights the approach which democratizes deep learning by making the technology accessible to smaller players in the field, such as startup companies and not-for-profits. The work appeared at NeurIPS 2021 and was reported on by Anil Ananthaswamy for Quanta Magazine.

🎙 Speaker:
Graham Taylor, Canada Research Chair and Professor of Engineering, University of Guelph

🎙 Moderator:
Wojciech Samek, Head of Department of Artificial Intelligence, @FraunhoferHHI

Join the Neural Network!
The AI for Good networking community platform powered by AI.
Designed to help users build connections with innovators and experts, link innovative ideas with social impact opportunities, and bring the community together to advance the SDGs using AI.

🔴 Watch the latest #AIforGood videos!

Explore more #AIforGood content:
1️⃣ AI for Good Top Hits

2️⃣ AI for Good Webinars

3️⃣ AI for Good Keynotes

📩 Stay updated and join our weekly AI for Good newsletter:

📅 Discover what's next on our programme!

🗞Check out the latest AI for Good news:

📱Explore the AI for Good blog:

🌎 Connect on our social media:

What is AI for Good?
The AI for Good series is the leading action-oriented, global & inclusive United Nations platform on AI. The Summit is organized all year, always online, in Geneva by the ITU with XPRIZE Foundation in partnership with over 35 sister United Nations agencies, Switzerland and ACM. The goal is to identify practical applications of AI and scale those solutions for global impact.

Disclaimer:
The views and opinions expressed are those of the panelists and do not reflect the official policy of the ITU.

#TrustworthyAI #AIforGoodWebinars

[berry]

A natural question is whether the is some computational gain, by doing the first training stage with the weight forecast, and then start a second stage of normal SGD optimization?!