Connections between physics and deep learning

Max Tegmark - MIT

[danodet]

I personally see a similarity between physics and deep learning in the way the world is made of encapsulated layers of realies. For example, as shown in thermodynamics, the macroscopic layer don't need to know every position and velocity of every particles. It only need to know certains computed features like temperature and pressure.

[jurvetson]

I think it's a fascinating summary of the tie between the power of neural networks / deep learning and the peculiar physics of our universe. The mystery of why they work so well may be resolved by seeing the resonant homology across the information-accumulating substrate of our universe, from the base simplicity of our physics to the constrained nature of the evolved and grown artifacts all around us. The data in our natural world is the product of a hierarchy of iterative algorithms, and the computational simplification embedded within a deep learning network is also a hierarchy of iteration. Since neural networks are symbolic abstractions of how the human cortex works, perhaps it should not be a surprise that the brain has evolved structures that are computationally tuned to tease apart the complexity of our world.

[WilliamBarksdale]

41:30 Opossum? Laughed out loud that this was the word that was used as an example, and even more amazing that someone from the crowd got it!

[williamkyburz]

the fact that his first neural network diagram is upside down is poetic ... classic Physics space cadet

[pafloxyq]

41:45 fractal nature of English!... Goddd that blows my head..!!!

[MattSiegel]

34:00 more advanced than my kindergarten :D

[subramanyam]

Sat through the whole lecture although most of it went above my head!!

[mikhailfranco]

The interleaving of linear evolution and non-linear functions is also how quantum mechanics works:
1. The propagation step is perfectly linear, conservative, unitary, non-local and time-reversible. It is a continuous wave with complex amplitude specified by the Schrodinger equation. There is no loss of information. There are no localized particles in this step. There is no space in this step.
2. The interaction step is discrete, non-linear, local and time-irreversible. It is a selection/generation/collapse of alternatives based on the Born Rule. There is a loss of information, as complex values are added and amplitudes squared to give non-negative real probabilities. The result is an interaction, the creation of space-time intervals from the previous interactions, identification of localized entities which might be called particles, and some outgoing waves that are correlated (entangled). Go to 1.

Einstein complained that the non-locality of QM was "Spooky action at a distance", but in the Quantum Gravity upgrade, space is only created by interaction, so it becomes "Spooky distance at an action".

[dr_flunks]

Max - loved your recent book: "Mathematical Universe" Good to see you have interest in ML!! Keep up the great work!!!

[spurious]

Max is absolutely brilliant, and a scientist of the absolute highest caliber, but his categorization of the different tasks within machine learning is incorrect. Modeling a joint probability, or p(x, y) is categorically referred to as a generative modeling, not unsupervised modeling, which is a different, though potentially overlapping, concept. Classification, correspondingly, is returning a class label for a given input, in standard notation, this is p(y|x). Prediction, or forecasting, is similarly p(x(t)|x(t-1), ....x(1)). Unsupervised learning, by contrast, does not have some conventional notation, it refers to a scheme where a class label y is not fed to the training system. The joint probability that he wrote for unsupervised learning actually says nothing about the presence or absence of supervision, unless the y is a label, in which case the formalism is just plain wrong.

I say this because there are lots of students looking at the work of brilliant scientists like Max, and they owe it to the students to have consistent and correct formalism, given that the students may still be learning.

[davidsewell]

I am having difficulty understanding step 11 of the paper in which Max goes from the taylor series expansion form of the activation function to the multiplication approximator. Does anyone know of a more detailed explanation of this?

[CHECKR]

Really cool talk Max. I'm curious who the audience for your talk was. I think they must have a really good foundation of knowledge in mathematics.

[eugenbarbula]

What is the difference between random forests and DNN?

[fiftyshadesofgrey]

Thank you very much, it was really helpfull. Even if I suggested smth like that idea before)

[godbennett]

Max is a bit 'late', for it has been known, for quite a while, of neural networks' compression-bound nature:



.

.

Albeit, we need subsume of larger problems, inclusive of Marcus Hutter's temporal difference aligned lemma, via hints from quantum mechanics, deep reinforcement learning (particularly deepmind flavoured) and causal learning (ie uetorch):



.

.



A code sample that initializes the confluence of temporal difference regime, abound the causal horizon:

[bobsalita]

Max shows why AI is a cross-disciplinary science. Really stimulating video.

[arunkannan]

Just after 7:05 whose talk is he referring to ? Ends with *ski.

[PauloConstantino]

He should definitely learn how to speak without those annoying pauses and weird mouth warps.

[chaoshi]

I haven't read the paper but the presentation is terrible. The examples are not clearly illustrated.