Yoshua Bengio | From System 1 Deep Learning to System 2 Deep Learning | NeurIPS 2019

Summary:

Past progress in deep learning has concentrated mostly on learning from a static dataset, mostly for perception tasks and other System 1 tasks which are done intuitively and unconsciously by humans. However, in recent years, a shift in research direction and new tools such as soft-attention and progress in deep reinforcement learning are opening the door to the development of novel deep architectures and training frameworks for addressing System 2 tasks (which are done consciously), such as reasoning, planning, capturing causality and obtaining systematic generalization in natural language processing and other applications. Such an expansion of deep learning from System 1 tasks to System 2 tasks is important to achieve the old deep learning goal of discovering high-level abstract representations because we argue that System 2 requirements will put pressure on representation learning to discover the kind of high-level concepts which humans manipulate with language. We argue that towards this objective, soft attention mechanisms constitute a key ingredient to focus computation on a few concepts at a time (a "conscious thought") as per the consciousness prior and its associated assumption that many high-level dependencies can be approximately captured by a sparse factor graph. We also argue how the agent perspective in deep learning can help put more constraints on the learned representations to capture affordances, causal variables, and model transitions in the environment. Finally, we propose that meta-learning, the modularization aspect of the consciousness prior and the agent perspective on representation learning should facilitate re-use of learned components in novel ways (even if statistically improbable, as in counterfactuals), enabling more powerful forms of compositional generalization, i.e., out-of-distribution generalization based on the hypothesis of localized (in time, space, and concept space) changes in the environment due to interventions of agents.

[evankim]

I like that Yoshua approaches the theory of neural networks in the language of probability at its core.

[AR-iutf]

Prof. Bengio is perhaps one of the key (if not the only key voice) who so clearly articulates in great detail what is lacking in DL to date and what could be one path forward ( and is kind enough to give links to all relevant references). Few exhibit the intellectual honesty and earnestness in helping the rest of us understand what to expect in the future.

Wish I had teachers like him when I went to school.

[gangfang]

It took me a month to fully understand everything he discussed in this presentation (at a high level). I think this is the future. Would love to hang out and discuss if anyone is in Toronto.

[CristianGarcia]

Yoshua: "Conscience is the next big thing"

Next job offering: AI Conscience Engineer

[araldjean-charles]

A big chunk of knowledge maybe pre verbal. Look at our cats, dogs, and other mammals.

[wehitextracellularidiombit]

Who's the speeker who introduced Mr YB? Is she a researcher too ?

[Dmdmello]

Isn't causality just a special case of correlation across time? At least that's how it seems to works for human intuition of causal effect, I think

if so, I don't see the fact that modern neural nets are only capable of learning correlations as an impediment for them to also learn causal relations.

[arjunashok]

The link for the slides don't work! Please update them!

[leo.budimir]

"In our community, the C-word (consciousness) ..." =D

[catsaresocute]

I am a horrible sister I just went to somome IS in the room and I just wanted to make sure He doesn't get into too much trouble what is like snnsnsjjsjsn Always make doubble an tripple sure that the absuive persons know a meeting has been argreed multiple Times and so that they can't deny it and schools so good for that too because it's so good that it can't be rejected socialy without going to tue relm of negelect- Like saying a sister May not teach her brother how to do things. Need to keep maybe book of Interactions w Jackob so I have a better case?