The Long Story of How Neural Nets Got to Where They Are: A Conversation with Terry Sejnowski

00:00 Start stream
5:26 SW starts talking
5:39 When did people first realize there were neurons in the brain?
6:30 Before the discovery of neurons, what did people think the brain was made of?
7:28 How did people figure out things in the brain had to do with electricity?
9:22 After discovery of electrical signals in the brain, where there huge developments of people studying and dissecting the brain?
10:47 When did people building machinery and people studying the brain connect?
13:17 The big moment for neural nets, logical structuring, the brain might be computing like a computer
18:47 Developments in the 40s and 50s- perhaps neural nets were original idea of what computers might be like
22:30 Developments in the 60s and 70s - Computers can prove theorems so what else can they do? Early development of machine translation projects.
29:09 Going back to McCulloch Pitts 1943 and idealized artificial neurons, Neuron structure and function in the human brain
35:04 Can artificial neural net weight matrices and biases capture characteristics of neurons and dendritic trees found in the brain?
47:26 Japanese advancements in neural networks in the 60s and 70s: Neocognitron, Linear predicates, Precursors of Hopfield networks
51:06 Early 60s resurgence of interest in artificial networks and intelligence.
1:03:07 Historical involvement of physicists in neural network development, communities coming together to figure out what was going on in the brain and how patterns of information can be stored in neural nets
1:18:50 Developments in the 80s and precursors to Boltzmann Machines
1:32:16 Developments in neural networks between the mid 1980s until 2011/2012
1:55:17 How did research change with the beginning of 2010s?
2:07:16 What's the next stage of neural nets? How is ChatGPT involved?
2:12:08 Looping Procedures and Learning Methods
2:17:28 Do you think there can be a human understandable theory of what you're seeing with dimensions and mathematics?
2:23:40 2012 breakthrough
2:26:19 Discussing project with Gerald Tesauro
2:23:27 What does solve the problem mean?
2:36:40 Was Dennis Gabor's 1959 'Electronic Inventions and their Impact on Civilisation' the first proposal of random circuits with weight tuning (to learn a black box function)?
2:40:28 What does he think of Robert Hecht Nielsen's Confabulation Theory as a top-down approach to the neocortex?
2:42:44 The HBP or Human Brain Project is worth Your attention in the future, It is amazingly complex plan to simulate how the Brain functions starting from simulating the Brain of the mice.
2:45:26 Terry mentioned that current models are about 3 or 4 orders of magnitude less compute than the brain - how many orders of magnitude in algorithmic development (backprop etc.) does he estimate we are?
2:49:42 One of the challenges with neural networks is that they can end up solving problems in ways that aren't intuitive to humans, and this results in issues with both trust and unexpected behaviour in edge cases or extrapolation beyond the training set. Is there much progress in "guiding" neural networks so they behave in a more intentional, human-like way?
2:51:42 What is the future? Reviewing the history.

Follow us on our official social media channels.

[shafqatullah]

Really great conversation! Eagerly looking forward to the next one on Future of AI/ML. Thanks @Stephen and @Terry!

[godynnel]

Great interview by Stephen here, best I've seen in terms of balancing giving airtime to the guest while also adding useful comments himself

[alberth]

excited to hear part 2 of this conversation: Back To The Future!

[wesleyhein]

This was an exceptionally interesting interview. Two really smart people talking about a fascinating set of subjects. I don't normally watch something 3+ hours in one sitting but that's what I did with this one.

[coolhead]

These are super informative. Thank you for sharing!

[taopaille-paille]

IT would be useful to have a few introductory words about the guest in the description for those that may not know him.

[SR-hmcf]

a trip down memory lane - and an intro to nn - great interview

[BillBenzon]

I sometime think of the brain as a "polyviscous" fluid. There are components with a very high viscosity, on the order of months and years, very low viscosity, on the order of milliseconds, and many levels in between. These components are all intermingled in the same physical space.

[rQuadrant]

If you came here out of an overwhelming sense of needing to catch up, you won't be disappointed. From the 3:04:31 mark: S. Wolfram, "But it so bizarre that the original neural net idea from whatever it is--85 years ago or something--is still, that's what is running inside Chat GPT." T. Sejnowski "It is! Shocking!"

Listen to the rest on a long walk to get filled in between now and 85 years ago.

[AlexShkotin]

great! is there any chance for timeline? thank you.

[Mentaculus]

1:30:07 → “Computing is now dirt cheap”❗️In Brian Greene’s interview with Brian Schmidt, Schmidt said that each “Training Iteration” cost $100s of MILLIONS to do for the LLMs! AND the latest are getting up to $400, 000, 000.

[charlesnutter]

AIl seemed so simple before the discussion; afterward, I am unsure! Great discussion.

[dr.mikeybee]

Your depth-interviewing skills are extraordinary, Stephen. And Terry has an amazing mind into which to delve!

[briancase]

This is an essential discussion; it should be viewed far and wide. Thanks! OMG, nettalk and dectalk. I think I vaguely remember those. I had no real interest in AI at the time, but nettalk made a big splash if I recall correctly. The Ridge computer's claim to fame at the time was that it was one of the first true commercial RISC architectures, so it bested the VAX in price/performance. There was a lot of minicomputer competition for the VAX at the time (Pyramid technology was another). Unfortunately, they didn't make the transition to a single chip fast enough. There were many computer companies that thought designing with SSI and MSI chips (small gate array and 7400 chips) would be competitive, but the window closed quickly (the Rational ADA machine was one example).

[omnipedia-tech]

Terry bringing up the alien language from Arrival at the end of the interview - Stephen had a bit of a twinkle in his eye, I think, because didn't his son help create the language? That's great!

[Jimserac]

This brings back such great memories. I was a programmer, and later software engineer, from 1972 and the IBM 360 Mainframe days, later taking a break to spend 3 semesters at U.R.I. taking courses in E.E., Computer Engineering and Advanced Math, 4 years at a company programming high precision industrial high precision Industrial scanning gauges in 8080 and Z-80 assembly language, going past the late 70's and some military work, through the 1980's programming mainly "C" for International Data Sciences in Lincoln, Rhode Island with my trusty copy of Kernighan and Ritchie on my desk (a young Mark Pesce sat in the cubicle next to me, he later moved to Australia) and finally 10 years at a financial services company programming an Annuities design system in C and C++ which ended in 2004. Along the way, in the mid 1980's, I would hang out at Brown Univ. where an enormous amount of talk and excitement revolved around neural networks, in particular Associative Networks and various other types. I went to a seminar run by Leon Cooper, physicist and Nobel prize winner involving a startup he was working on for intelligent traffic light control if I recall correctly. The feeling was that a breakthrough was imminent. We studied Bart Kosko whose mathematics were impressive but then got lost in a maze of "Fuzzy Logic" and talk about the Japanese 5th Generation project which seems to have created a storm of expectations but ended more like Shakespeare's "full of sound and fury, signifying nothing".
In 2004, my job having been outsourced to India I could see that "ancient geeks" in their mid to late 50's were of no interest to companies and switched careers, entering a College of Oriental Medicine in Florida for a whole new set of adventures which persist to this day in my quest to learn Qing era Chinese.
So a talk of this kind, which appears to fill in the gaps from the mid 1980's to the exciting stuff happening now, is of interest tome. Many thanks for the timeline breakdown !!

[yoyo-jcqg]

The 1990's is not a decade i would just gloss over, because both big data and processing power were crucial to neural net advancement, and the 90's brought us the internet, intel's major breakthrough CPU the pentium pro 64 bit, and Nvidia's first GPU a coprocessor that specializes in computationally intensive tasks.

[silberlinie]

No, of course we can only start to stop the accumulation of
cumputing nodes when we have reached the order of
magnitude of the numbers of the natural model - our brain.
2:01:42

[grfdeadfg]

Why The Fuck Was This In my "Relaxation & Sleep" Playlist...

[BillBenzon]

Chomsky was irrelevant to computational linguistics. Syntax was based on dependency grammar not phrase structure. The worlds of AI and MT (machine translation) were separate well into the 1970s and 1980s. Different communities, journals, and confereneces.