Adagrad Algorithm Explained and Implemented from Scratch in Python

Adagrad is an often used extension of stochastic gradient descent that work well for sparse parameter space like text or images. In this video I'll explain and show you how to implement it!

Credit to : Max Olson for the picture in the thumbnail, sorry I to have cut the watermark in the picture. The faint background music is from Youtube Music!

The implementation is very straighforward once the cumulative sum of gradient is understood as it is an extension of the stochastic gradient descent.

Here is a definition of adagrad from wikipedia:
"AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011. Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse. This strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative. Examples of such applications include natural language processing and image recognition. It still has a base learning rate η, but this is multiplied with the elements of a vector {Gj,j} which is the diagonal of the outer product matrix."

----

----
Follow Me Online Here:

___

Have a great week! 👋

[captainamerica]

What a thorough explanation. It really helps me a lot.

[Lorenzo]

thank you very much for the tutorials and code!
But I don't quite understand why both AdaGrad and AdaDelta perform poorly for these examples?

[auraSinhue]

The first link in the description is broken