Neural Ordinary Differential Equations

This talk is based on the first part of the paper "Neural ordinary differential equations". Authors introduce a concept of residual networks with continuous-depth, what they consider as ordinary differential equations (ODEs). Correspondingly, inputs of neural networks are considered as an initial state of ODEs, and outputs as a solution obtained by ODE solver. One of the main advantages of such approach is the constant memory cost with respect to the model depth. However, training of such networks requires introduction of adjoint function (standard technique from optimal control theory). One of the curious points is that solving of ODEs for the adjoint function can be considered as continuous analog of backpropagation.

[mingli]

This video relly helps me a lot, They provide the details about how to get the ODE for caculating the gradient of Loss function to params, which is key part of this paper

[ranam]

Please sir tell me neural differential equations are special