AMMI Course 'Geometric Deep Learning' - Lecture 11 (Sequences & Time Warping) - Petar Veličković

Video recording of the course "Geometric Deep Learning" taught in the African Master in Machine Intelligence in July-August 2021 by Michael Bronstein (Imperial College/Twitter), Joan Bruna (NYU), Taco Cohen (Quallcom), and Petar Veličković (DeepMind)

Lecture 11: Static and dynamic domains • Recurrent Neural Networks • Translation invariance • Time warping • Warping the ODE • Discrete Warped RNNs • Gated RNNs • LSTM

[TheAIEpiphany]

Nice lecture Petar! Some questions/comments:

30:00 The maths looks enticing but very hand-wavy - inputting delta=1 may induce a huge error into the approximation, effectively h would have to be linear for a "long" duration for this to hold. (38:55 clarifies this - if time is dilated then effectively delta becomes less than 1 (depending on the amount of dilatation) - leaving this comment in case someone gets confused here as I did)

30:40 Is it time-warping invariant or equivariant here? h(tau(t)) = h(t) would be time-warping invariance, right? It looks more like equivariance.
(48:00 addresses this but it still looks like equivariance)

31:05 Not completely clear why we need a derivative w.r.t. t and not tau(t)?

38:45 Soo they are not really time warp invariant (equivariant?) they are dilated time warp invariant (equivariant?) :)) Does this make any practical difference? Could you always assume that t was very contracted and then anything you sample is a dilated version of it? Is that the idea?

43:10 Do the additional gating mechanisms render LSTM non time warp invariant? It's not quite obvious that just because it has more mechanisms it preserves the (dilated) time warp invariance (equivariance) of its simpler gated brothers. :)