Lecture 3 Part 2: Finite-Difference Approximations

MIT 18.S096 Matrix Calculus For Machine Learning And Beyond, IAP 2023
Instructors: Alan Edelman, Steven G. Johnson

Description: Finite difference approximations are useful tools, e.g. to check analytical derivatives, but lead to numerical analysis to understand truncation and roundoff errors

License: Creative Commons BY-NC-SA

[hannahnelson]

Ok takeaways from this.

1. Finite differences for small perturbations are what derivatives approximate and thus a good way to check them.

2. By dividing the error of the approximation eg; the difference between the finite perturbation actual difference and the predicted derivative distance. By a term linear to dX (I see no reason that this should not just be dX itself) we can get a relative error term. The order of this relative error term is tied to the order of the error. Though I do not understand why if we are going to identify the order of the relative error why we would not just identify the order of the original normed error instead. It honestly seems simpler.

3. Round off approximations occur below a certain threshold due to the nature of floating point numbers.

4. It seems implied but unstated that the rounding threshold at which the prediction becomes worse is determined by the order of the error. This could perhaps be leveraged to identify the order of the error simply rather than using a complicated fitting method.

This part appeared implied as if it was either obvious or will be taught at a later date.

[MayankGoel]

I initially thought I knew finite differences well and there isn't much to learn. I am glad I was wrong

[hannahnelson]

19:00 I don't think I understand why we divide by the exact. Wouldn't this break entirely if the exact is actually 0?

[hannahnelson]

I feel that it would be more reasonable do divide by dA. Is there some technical reason why we divide by exact?

[zwzw]

Alan goes too fast, Steven goes too slow