ContinualAI RG 'Efficient Continual Learning in Neural Networks with Embedding Regularization'

[24-07-2020] ContinualAI Reading Group "Efficient Continual Learning in Neural Networks with Embedding Regularization"

Abstract: Continual learning of deep neural networks is a key requirement for scaling them up to more complex applicative scenarios and for achieving real lifelong learning of these architectures. Previous approaches to the problem have considered either the progressive increase in the size of the networks, or have tried to regularize the network behavior to equalize it with respect to previously observed tasks. In the latter case, it is essential to understand what type of information best represents this past behavior. Common techniques include regularizing the past outputs, gradients, or individual weights. In this work, we propose a new, relatively simple and efficient method to perform continual learning by regularizing instead the network internal embeddings. To make the approach scalable, we also propose a dynamic sampling strategy to reduce the memory footprint of the required external storage. We show that our method performs favorably with respect to state-of-the-art approaches in the literature, while requiring significantly less space in memory and computational time. In addition, inspired inspired by to recent works, we evaluate the impact of selecting a more flexible model for the activation functions inside the network, evaluating the impact of catastrophic forgetting on the activation functions themselves.

The speakers for this reading group were:

• Jary Pomponi

------------------------ ContinualAI Reading Group ------------------------

The ContinualAI Reading Group is hosted every Friday in collaboration with MILA and it is based on roughly 60 minutes discussion about a particular Continual Learning paper. Occasionally, a speaker is invited to explain his work. Join our reading group on slack! It will be the occasion to talk with other researchers of this amazing field!

------------------------ About ContinualAI ------------------------

ContinualAI is an official non-profit research organization and the largest open community on Continual Learning for AI. We aim at connecting people and working better together on this fascinating topic we consider fundamental for the future of AI.

It's thanks to people like you that we are making ContinualAI a reality!