Regularization (C2W1L04)

Follow us:

[manuel]

Clarification about Regularization

Please note that at 5:45, the Frobenius norm formula should be the following:

nˡ n[ˡ⁻¹]
||w[ˡ]||² = ⅀ ⅀(w[ˡ]ᵢ, ⱼ)²
ᵢ₌₁ ⱼ₌₁

The limit of summation of i should be from 1 to n[ˡ],

The limit of summation of j should be from 1 to n[ˡ⁻¹],

(it's flipped in the video). The rows "i" of the matrix should be the number of neurons in the current layer n[ˡ];

whereas the columns "j" of the weight matrix should equal the number of neurons in the previous layer n[ˡ⁻¹].

[exampreparationonline]

Adding regularization will help to prevent over-fitting in the neural network.


L2 regularization is most preferred as it doesn't reduces the weight to zero, it minimizes the weight. L1 is preferred when we want to compress our model.

weight decay, dropout, and early stopping are few measures to be used as regularization

[xiaodong]

I think the dimension of w[l] is ( n[l], n[l-1] )

[sumitvaise]

Very insightful. Lots of concepts in 9 mins.

[aliciaornelas]

In minute 6:20 the dimensions of weights are wrong

[vaibhav]

W[l] would be a (n[l], n[l-1]) matrix I guess, not a (n[l], n[l-1]) mat.

[cpqhucn]

If we add bias regularization, won't we essentially set it to be equal to zero because it would be the most 'optimal' decision in terms of loss minimization?

[banipreetsinghraheja]

Wrong Notation of W vector, and hence the sigma terms!

[Troglodyte]

I would love to know the "really arcane reasons".

[sayantanmazumdar]

Doubt :-

If there was l1 regularization, then would the formula be

dwl=dwl+(lambda/2m) * (wl)/(|wl)

I think my answers would be right as per the derivatives, but i still l want to check it . Can someone help me out ?

[brodie_brodes]

What is the mathematical term and meaning for the funny looking R symbol at 1:08?

[AvinashSingh-bkkg]

Why we always focus on decreasing the weights? We already consider small random weights.

[agentanakin]

If you can create a slide, why then add illegible handwriting? I stopped watching at 0:45.