Adjoint Sensitivities of a Linear System of Equations - derived using the Lagrangian

We can also arrive at the equations for the adjoint sensitivities of a linear system using a different point of view. Here, we frame it as an equality-constrained optimization problem. Then, we can build a Lagrangian of the problem, which total derivative is identical to the sensitivities, we are interested in. The involved Lagrange Multiplier can be chosen arbitrarily, since the primal feasible already constraints our minimum. So, let's choose in a way such that we avoid the computation of a difficult quantity. And that's all the magic! :) After some more manipulation, we arrive at the same equations as in the previous video.

-------

-------

Timestamps:
00:00 Introduction
00:49 Similar to using implicit differentiation
01:15 Implicit Relation
01:48 Dimensions of the quantities
02:26 Lagrangian for Equality-Constrained Optimization
03:37 Total derivative of Lagrangian
05:02 Gradient is a row vector
07:31 The difficult quantity
08:33 Clever Rearranging
09:27 Making a coefficient zero
10:31 The adjoint system
12:01 The gradient is now easier
12:37 Total derivative of Loss
14:35 Strategy for d_J/d_theta
15:47 Scales constantly in the number of parameters
16:27 The derivatives left in the equation
17:01 Outro

[derludwig]

Thank you for the nice Video! Well structured an easy to follow!

[Pengzhongluo]

Thanks so much for the video explanation! 😍
My understanding is that both Lagrangian and implicit derivative methods yield the same sensitivity analysis results, right?
The Lagrangian approach is more convenient for large linear equation constrained implementations, as it requires only one forward and one backward propagation computation per iteration. In CFD, sensitivity analysis and adjoint optimization are typically implemented using adjoint equations, right?

[GordenMax-ty]

Thank you very much for the nice video. Could you recommend some textbooks for adjoint methods for sensitivity analysis?

[蕾蕾-dc]

Hello, teacher. Your vedios benifit me a lot ！And I have a question，It has been bothering me for a long time. How can something be called adjoint?

[陈金彬]

thanks for this high quality sharing! there is a question: why is the adjoint equation represents the "backward" process, how does it preform in the adjoint equation

[usefulknowledge]

I have a question regarding the minimization of the Lagrangian. When it comes to minimization of the Lagrangian function, we usually have to first maximize the value of lambda, and then minimize the resulting function (which contains the max value of lambda), to get solution. If lambda is not maximized, the solution that we get is a lower bound to the actual solution. In the case here, where you choose the value of lambda to avoid calculating one of the derivatives, will this guarantee that the solution we get is the same solution as the one where we maximized lambda?

[icojb]

Thanks for the videos 😄- Question, isnt the dependence of J() on theta implicit through x? Why in the problem statement then do we include theta ... ? Thanks

[orjihvy]

Do you also have plans to cover Gaussian Process at some stage?