Beyond the Patterns 28 - Petar Veličković - Geometric Deep Learning

0:00 Welcome Address
2:43 The Erlangen Programme
7:46 Geometric Deep Learning - Introduction
8:04 Learning in High Dimensions is Hard
10:14 Symmetries, Groups, and Invariances
23:17 The Blueprint of Geometric Deep Learning
27:29 The "5G" of Geometric Deep Learning
29:25 Graphs
42:06 Grids
48:19 Groups
57:24 Geodesics and Gauges
1:01:53 Discussion
1:13:31 Wrap up & Closing

It’s a great pleasure to welcome Petar Veličković from Deep Mind to our Lab!

Abstract: The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods. Indeed, many high-dimensional learning tasks previously thought to be beyond reach –such as computer vision, playing Go, or protein folding – are in fact feasible with appropriate computational scale. Remarkably, the essence of deep learning is built from two simple algorithmic principles: first, the notion of representation or feature learning, whereby adapted, often hierarchical, features capture the appropriate notion of regularity for each task, and second, learning by local gradient-descent type methods, typically implemented as backpropagation.
While learning generic functions in high dimensions is a cursed estimation problem, most tasks of interest are not generic, and come with essential pre-defined regularities arising from the underlying low-dimensionality and structure of the physical world. This talk is concerned with exposing these regularities through unified geometric principles that can be applied throughout a wide spectrum of applications.

Such a ‘geometric unification’ endeavour in the spirit of Felix Klein’s Erlangen Program serves a dual purpose: on one hand, it provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand, it gives a constructive procedure to incorporate prior physical knowledge into neural architectures and provide principled way to build future architectures yet to be invented.

Bio:Petar Velikovi is a Senior Research Scientist at DeepMind. He holds a PhD in Computer Science from the University of Cambridge (Trinity College), obtained under the supervision of Pietro Liò. His research interests involve devising neural network architectures that operate on nontrivially structured data (such as graphs), and their applications in algorithmic reasoning and computational biology. He has published relevant research in these areas at both machine learning venues (NeurIPS, ICLR, ICML-W) and biomedical venues and journals (Bioinformatics, PLOS One, JCB, PervasiveHealth). In particular, he is the first author of Graph Attention Networks—a popular convolutional layer for graphs—and Deep Graph Infomax—a scalable local/global unsupervised learning pipeline for graphs (featured in ZDNet). Further, his research has been used in substantially improving the travel-time predictions in Google Maps (covered by outlets including the CNBC, Endgadget, VentureBeat, CNET, the Verge and ZDNet).

Geometric Deep Learning Website:

Michael Bronstein's Blog Post on Geometric Deep Learning:

Petar's Talk at Cambridge:

Music Reference:
Damiano Baldoni - Thinking of You (Intro)
Damiano Baldoni - Poenia (Outro)

Music Reference: 
Damiano Baldoni - Thinking of You (Intro)
Damiano Baldoni - Poenia (Outro)

[TheAIEpiphany]

Great talk! The group convolutions finally clicked for me after your discussion (48:27 - 57:26)! :))

One comment on my side: 54:20 this dichotomy in Spherical CNNs seems a bit inelegant, every other model you described is uniform in this respect i.e. we just keep on stacking the same types of layers. Any thoughts on this?

[raghavamorusupalli]

Geospatial guys need this model very badly. I applied this concept for image inpainting sometime back.

[memorylane]

great presentation!- giving us useful ideas for portability.
- The core idea of g -->G, being matrices that represent most Deep Learning Models (linear + activation).
- But the symmetricity of g doesn't give us a measure of portability across various g's though..
Meaning, the data U they are trying to map are still different probability distributions.
(example, a learnt latent vector of human face different from Cars).
We are looking into model portability, for better reuse, faster generation.
So, maybe look at identifying g's that are closer to each other, with some kind of geometrical tests (thinking out loud)...

[lololololalso]

Thank you !!! I really wonder how GDL will end up with Gestalt principles for computer vision problems.

[aaryanbhagat]

Thanks for this nice upload but it would be more influential if this would be timestamped (same for the future videos).

[manncodes]

Hey, can we have the copy of the slideshow if it's possible?

[chifortudor]

Is there a link to a digital version of the book available ?

[JethroDjan]

Hello. I am a math person and I get the maths aspect of the talk. But I know next to nothing about deep learning ( or even machine learning for that matter apart from playing around with Tensorflow). Can someone direct me as to where I can start to learn this from a theoretical perspective? I want to do something on this later for thesis... Thanks!