I've Been Doing This Wrong The Whole Time ... The Right Way to Save Models In PyTorch

It turns out I wasn't saving PyTorch models correctly. You really need to save the optimizer state as well as the state of the current weights of your deep neural network. This is critical to getting deep Q learning and actor critic agents training on complex environments that may require multiple sessions spread over time.

Starter code for this video is here:

Learn how to turn deep reinforcement learning papers into code:

Get instant access to all my courses, including the new Prioritized Experience Replay course, with my subscription service. $29 a month gives you instant access to 42 hours of instructional content plus access to future updates, added weekly.

Or, pickup my Udemy courses here:

Deep Q Learning:

Actor Critic Methods:

Curiosity Driven Deep Reinforcement Learning

Natural Language Processing from First Principles:

Just getting started in deep reinforcement learning? Check out my intro level course through Manning Publications.

Reinforcement Learning Fundamentals

Here are some books / courses I recommend (affiliate links):

Come hang out on Discord here:

[forough]

Many thanks for updating us. I really appreciate you providing us with such wonderful courses.

[adamjiang]

Hey Phil, how to save/reload the seed for numpy, env.seed, env.action_space.seed, T.cuda.manual_seed_all and other related seed setting? Just say if I need to run a long training, but have to switch over to different machines (very likely happen in HPC). Could you please provide some advice on this? Thanks in advance.

[jose-alberto-salazar-jimenez]

Thanks for update. I wonder how would you go about loading the model, once trained, for "testing"? I've tried for example, loading the model state to q_eval, setting q_eval to ".eval()" mode, using "with torch.no_grad()", then getting the predictions with "model(observation)", but the model/agent doesn't perform as how it was doing during training (it does really bad in comparison).

[forough]

Dear Phil,
May I ask why q_next[done] = 0? Is not q_next a single value?

[meowatthemoon]

Hi, thanks for the video. Can you explain what are the benefits of saving the state of the optimizer?

[markadyash]

hey phil good to see you
what you working on RL or anything else?

[MrCmon]

I did horrible on all of this organizational shit for by bachelor thesis, next time I wanna be organized from the start. Gonna watch what a bunch of different experienced people are doing.

[killereks]

What are ur pc specs? Training is rly fast

[fastaalemyapanadam]

Thanks for video.Please check your e mail