Talk: Assessing Limitations of Three-Factor Hebbian Learning Relative to Deep Reinforcement Learning

Speaker: Jordan Lei, University of Pennsylvania (grid.25879.31)
Title: Assessing Limitations of Three-Factor Hebbian Learning Relative to Deep Reinforcement Learning
Emcee: Doby Rahnev
Backend host: Yang Chu

Presented during Neuromatch Conference 3.0, Oct 26-30, 2020.

Summary: Three-Factor Hebbian Learning (also known as Reward-Modulated Hebbian Learning) is an important framework for neuroscientists because it provides a biologically relevant framework for modeling learning in the brain. However, the effectiveness of Three-Factor Learning, which specifies that synaptic plasticity is driven only by presynaptic activity, postsynaptic activity, and a global neuromodulator, is limited. By contrast, Deep Reinforcement Learning (DRL) provides more computationally powerful learning rules that typically use an additional factor related to the influence of the synapse or neuron towards the action taken by the agent. However, the biological relevance of these rules is less well understood. The primary goal of this project is to better understand the specific network and task conditions that cause Three-Factor Learning agents to fail, and the specific features of DRL algorithms that can overcome these problems and thus could potentially have relevance to learning in the brain.

We consider four primary challenges that the brain must overcome to learn effectively. The first is adding uncertainty to the time of stimulus onset on each trial, which is typically used in real-world learning tasks but less often in simulations. The second is adding multiple stimulus dimensions and changing which one is relevant for learning at a given time. The third is network complexity, including simply moving from a simple linear network that is used in many simple modeling exercises to a multi-layer network needed to solve more complex problems. The fourth is additional forms of plasticity, such as a perceptual learning-like mechanism that can affect the input to the network, that must interact synergistically with other learning in the network to produce appropriate outputs. Preliminary results suggest that each of these challenges are overcome more effectively by DRL algorithms than Three-Factor Hebbian rules, implying a potential role for those algorithms in the brain.