L12.4 Adam: Combining Adaptive Learning Rates and Momentum

-------

This video is part of my Introduction of Deep Learning course.

-------

[sergeykurk]

With general idea of adaptive learning rates, how do we determine whether we're going to the right or wrong direction? So we know whether we need to add "B" to the local gain or multiply by it (1-B)

[danielgurgel]

Hi, great course. The link for L12.5 is wrong on your website (blog).

Thank you very much,
I have some question. In my task the number of features can be changed, so I have to develop the NN model where the first layer is updated at runtime, but others don't (something like transfer learning, but without freezing other layers). Then, should I initialize the optimizer again? The part of the code I wrote below.
Thank you.

second_layer_input_size = int(model.fc1.out_features) # input number in the second layer
model.fc1 = nn.Linear(new_features_len, second_layer_input_size) # updating neural network's first layer at runtime for more features
model.to(device) # Again to device, without it, it doesn't work. I'm not sure does it mean that I lost already trained weights ?
optimizer = optim.Adam(params=model.parameters(), lr=learning_rate) # Should I do it ???
features_len = new_features_len

[hungerbear]

Nice tutorial, thx for sharing! In your experience, is it redundant to use learning rate scheduler with Adam?

[RayGuo-bonr]

I'm learning ML. I found most videos spent a lot time on explain complex "equation" . Actually those equations are easy to understand if you pass in some numbers to explain.🙂Not sure what happened in these area.