ECCV 2022 Tutorial: Deep Energy-Based Learning in Computer Vision

This tutorial provides a quick introduction of current deep energy-based modeling and learning methodologies. It starts from the background of energy-based models from the perspective of computer vision, and then presents three categories of deep energy-based frameworks, including deep energy-based models in data space, energy-based cooperative learning frameworks, and energy-based models in latent space. This tutorial aims to enable researchers to learn about the current advance of deep energy-based learning and apply the knowledge to other domains.

[oleksiygrechnyev]

A very interesting lecture.

It's definitely not beginner-level. A lot of material is presented in a highly condensed form.
Prior knowledge in both Langevin dynamics and energy based models is a must.

[jianwenxie]

You also can check videos of the previous EBM tutorials below:

CVPR 2021 Tutorial on Theory and Application of Energy-Based Generative Models

Part I: Fundamentals

Part 2: Advanced

Part 3: Applications

ICCV 2021 Tutorial on Theory and Application of Energy-Based Generative Models


IJCAI 2022 Tutorial on IJCAI 2022 Tutorial: Deep Energy-Based Learning

[jianwenxie]

Part I : Background

- Knowledge Representation: Sets, Concepts and Models
- Pattern Theory
- Texture Modeling
- Clique-Based Markov Random Field
- FRAME (Filters, Random field, And Maximum Entropy)
- Inhomogeneous FRAME Model
- Sparse FRAME Model
- Hierarchical Sparse FRAME Model
- Deep FRAME Model
- Deep Energy-Based Models – Generative ConvNet
- Three Research Directions of Deep Energy-Based Learning

Part II : Deep Energy-Based Models in Data Space

- Maximum Likelihood Estimation of Generative ConvNet
- Mode Seeking and Mode Shifting
- Adversarial Interpretations
- Short-run MCMC for EBM
- Multi-Grid Modeling and Sampling
- Multi-Stage Coarse-to-Fine Expanding and Sampling
- Energy-Based Image Inpainting
- One-Sided Energy-Based Image-To-Image Translation
- Patchwise Generative ConvNet for Internal Learning
- Spatial-Temporal Generative ConvNet: EBMs for Videos
- Generative VoxelNet: EBMs for 3D Voxels
- Generative PointNet: EBMs for Unordered Point Clouds
- Energy-Based Continuous Inverse Optimal Control

Part III : Deep Energy-Based Cooperative Learning

- Generator Model as a Deep Latent Variable Model
- Maximum Likelihood Learning of Generator Model
- Two Generative Models: EBM vs. LVM
- Cooperative Learning via MCMC Teaching
- Cooperative Conditional Learning
- Cycle-Consistent Cooperative Network
- Generative Cooperative Saliency Prediction
- Cooperative Learning via Variational MCMC Teaching
- Cooperative Learning of EBM and Normalizing Flow

Part IV : Deep Energy-Based Models in Latent Space

- Latent Space Energy-Based Prior Model
- Learning by Maximum Likelihood
- Prior and Posterior Sampling
- Learning and Sampling Algorithm of Latent Space EBM
- Conditional Latent Space EBM for Saliency Prediction