Slopes of Machine Learning - Computerphile

Coding Partial Derivatives in Python is a good way to understand what Machine Learning "secret sauce" has to do. Professor Thorsten Altenkirch explains what it means, and then demos in Python.


This video was filmed and edited by Sean Riley.

[jamie_ar]

Love anything with Professor Thorsten, his accent is awesome!

[AgentM]

Computer scientist looking out of the windows:
I see Trees! I see Tree(3)'s!

[leftaroundabout]

To elaborate a bit on what he said at the end about _efficiency when there are thousands of parameters_ (instead of just a handful):
The algorithm presented here is _symbolic differentiation_ – very inefficient because it actually generates a tree for each partial derivative. Now, that can easily be improved by right away evaluating it the result _numerically_ instead, whilst doing the derivative calculation (basically, you actually return `0` and `1` instead of `Const(0)` and `Const(1)`). That is then called forward-mode automatic differentiation, but it's still inefficient when there are lots of parameters because for each partial derivative you must traverse the entire input expression, giving a quadratic time-complexity. The real trick is, instead of going through all the partial derivatives (which can be interpreted as sending in all possible tangent-vector inputs), you send in all possible co-tangent vectors at the end of the function, and calculate the corresponding co-tangent vector at the input. This is then called reverse-mode AD or “backprop”. Because the function is typically a scalar-valued cost function, that means you only need to send in _one_ co-vector at the end, thus the complexity scales only linearly in the number of parameters.

In addition to these algorithmic issues: Python on a CPU is very slow. To get state-of-the-art performance, the low-level code is generally written in Cuda to run on a GPU, though most users don't really need to care because they can just call a ready-optimised libary from Python.

[andrewwmitchell]

"Is this going to be Numberphile territory?"
"No... Maybe... It's borderline."
hahahahah

[MegaRad]

Calculus: elegant, curvy equations
Python: brackets

[wrld]

Came for machine learning, stayed for the accent

[deadalonethe]

He's a lovely dude, but if PEP 8 could have a question to him, it would be: Am I a joke to you?

[TheKhashix]

I always have to watch this guys videos many times over to fully comprehend it. It might not be as daunting to a computer scientist, but to a humble web developer it’s quite a bit. And greatly appreciated!!!

[Seegalgalguntijak]

Oh Mann, Thorsten, Dein deutscher Akzent ist echt der Hammer. Die Amis nennen sowas dann "badass" ;-)
Aber zum Thema: Sehr interessant!

[qzbnyv]

pandoc is great. Both as a program and the Haskell library itself

[gregf]

Yay for pandoc! Fantastic that you call it out. Thanks :)

[liqo]

Would love to hear more type theory and constructive mathematics from Professor Thorsten!

[NerdyRodent]

Nice. I’m playing with StyleGAN2 at the moment and all this gradient descent stuff is interesting 🙂

[allan.n.]

Echt Hammer.. ! Bitte ..More videos with Thorsten 😎🤓

[NikolajLepka]

Please oh please follow PEP 8 when writing Python code.
There's a standardised style guide for a reason.
Simple things like spaces around operators and no floating colon

[StreuB]

Prof. Thorsten has lost a lot of weight. He looks very healthy as well!! On top of it, GREAT talk!!!! I just finished Calc3 this past semester so this was right up my alley!!

[franklinvp]

5:57 "is not a division". That comment that is neither the entire truth not entirely false, but everyone likes to make about Leibniz's notation. And then 8:28 ... "and then we divide" :D

[andersjohnson]

I think the parameter name in Plus.eval and Mult.eval should be "env" rather than "val". It works anyway because it can access "env" as a global. Did I get that right?

[DasGrosseFressen]

OMG, CS really needs to start making standard notations... They have many examoles of the same concept with multiple names... "Recursive deacent" yeah sure ooorr "gradient descent"

[black_platypus]

10:50 Rodents rejoice!
Prof. Altenkirch will feed you when you're hungry!
With him, you'll always get a bit of a mouse full :3