Nonlinear System Solve - Pushforward/Jvp rule

Next to the numerical solution of differential equations, you also find nonlinear solvers for a bunch of other applications like nonlinear optimization problems or, more recently, deep equilibrium problems. Being able to propagate tangent information is a crucial component in order to get a differentiable chain of computations to then use forward-mode Automatic Differentiation to obtain derivative estimates.

For implicitly given problems, like the roots of nonlinear systems of equations, there are two approaches: One can either differentiate through the solver (also called unrolling or piggybacking) or differentiate over the solver by employing the implicit function theorem and thereby deriving a custom forward rule. Often times, the latter is also the only way of doing so, as these numerical kernels tend to be implemented either in a non-differentiable fashion or rely on non-differentiable third-party libraries.

-------

-------

Timestamps:
00:00 Nonlinear System Solving as a function
01:00 Applications
01:15 Solution by e.g. Newton Raphson
01:53 Dimensionalities involved
02:37 Task: Forward Propagation of tangent information
03:22 Without unrolling by the forward-mode AD engine
04:41 General Pushforward/Jvp rule
06:36 Total derivative of optimality criterion/zero condition
08:15 Identifying the (full and dense) Jacobian
09:15 Plug Jacobian back into general pushforward/Jvp expression
10:20 Requires solution to a LINEAR system of equations
10:56 Full Pushforward rule
11:49 How about the additional derivatives?
12:15 Finding right-hand side with a Jacobian-vector product
12:36 Solve linear system matrix-free Jacobian-vector product
14:03 Summary
16:00 Outro