What is Automatic Differentiation?

Errata:
At 6:23 in bottom right, it should be v̇6 = v̇5*v4 + v̇4*v5 (instead of "-").

Additional references:
Griewank & Walther, 2008: Evaluating Derivatives: Principles and Techniques

Alleviating memory requirements of reverse mode:
Griewank & Walther, 2000: Algorithm 799: revolve: an
Dauvergne & Hascoët, 2006. The data-flow equations of checkpointing in
Gruslys et al., 2016: Memory-efficient Backpropagation

Example software libraries using various implementation routes:

Source code transformation:

Operator overloading:

Graph-based w/ embedding mini lanugage:

Special thanks to Ryan Adams, Alex Beatson, Geoffrey Roeder, Greg Gundersen, and Deniz Oktay for feedback on this video.

Music: Trinkets by Vincent Rubinetti

Links:

If you'd like to help support the channel (completely optional), you can donate a cup of coffee via the following:

[anjelpatel]

I like how more and more people are adopting 3b1b's style. Makes the content much better and easier to understand. This slowly converts a lot of the more complicated topics into easy-to-digest modules.

[raminbohlouli]

I knew basically 0 about AD and didn't know where to start since all the articles, websites, books etc that I have looked into, explained everything in a really comlicated way. I would like to thank you immensly for this very informative yet simple video! Now I know enough to dive deeper into the concept. This video was all I needed. Keep up the great work! You got yourself a new follower.

[DaveJ]

Automatic Differentiation, both forward and backward, plus mixed mode and the Hessian example at the end, all in less than 15 minutes, and totally clear.

Great content.

[arnold-pdev]

Went from complete ignorance to understanding in 15 min. Thank you!

[arkasaha]

Man this is pure gold. We all use this stuff but hardly have a clear idea about it's nitty-gritties. Thanks for thre awesome content and presentation, keep it up! :)

[stathius]

Class act, being concise and clear at the same time is no easy feat. Thank you.

[tom-sz]

Great video! Where can I learn more about the rounding and truncation errors plot at 2:06? I need to make an analysis of these errors for a project. Thanks :)

[jorgeanicama]

One more note ARI. I think there is a small typo. From minute 7:36 until 7:46 the derivative of V6 should be a "+" instead of a "-".

[chandank]

Your way of explanation is from india sir♥️

[TheLokiGT]

Very good job. One of the very few good videos I've seen around about autodiff.

[esaliya]

This is a neat summary that's hard to find in a single place!

[jorgeanicama]

Thank you Ari. I used symbolic computation in the past but this novel way of calculating derivatives is quite interesting. Learnt lots by watching your video. For sure, I will follow up with the recommended literature

[jaf]

Well done, superbly explained in context of other differentiation methods. Exactly what I needed!

[aronferencz]

At 6:37, when you show the calculations of the primals and the tangents shouldn't the last derivative v'_6 be equal to v'_5*v_4 + v_5*v'_4, rather than v'_5*v_4 - v_5*v'_4 by the product rule for differentiation?

[koushik]

This is highly motivated by Andrej Karpathy's lecture, but very clear explanation. It is indeed a good addition to my resource list.

[stansilverman]

In order to explain this to my wife, I differentiated voter rights-the analog process humans decide who should be allowed to vote, someone who looks like me, or everyone?. I think she got it. Brilliant Ari

[abhishek.shenoy]

This is so well explained! I love the quality of your videos!

[VHenrik]

Just as a note for anyone wondering, the arxiv link doesn't work because it includes the closing parenthesis. Otherwise great video!

[SohailKhan-zbtd]

Thanks a lot. This kind of videos are really a lot of hardwork to produce. Thanks a lot

[PahenPWNZ]

Awesome explanation, thanks!
But I still have one question, can someone explain please, at 12:05, right column (Adjoints)
I don't understand how did we get these values (f. e. v bar 5 = v4 * v bar 6, etc...) From where did these values come from?
If we use the formula at the previous slide with sum of children nodes, I get different values..