7 PyTorch Tips You Should Know

Here are 7 tips for improving your PyTorch skills. These are all things that I thought of because I use on a normal basis. PyTorch has a lot of need things you can do with modeling to distributions, let me know other tips you have in the comments below!

Tips:
1. Create tensors directly on the target device
2. Use Sequential layers when possible
3. Don't make lists of layers
4. Make use of distributions
5. Use the detach method when the gradient is not needed
6. How to delete a model from the GPU
7. Call the eval method before testing

[kuretaxyz]

You can also initialize Sequential layer with an OrderedDict if you want to access the layers with names instead of list indices.

[snippletrap]

Converting a list of layers to a Sequential module is not always best. For example, suppose you want to add positional embeddings to each intermediate output. Can't do that with Sequential. But you can use a ModuleList as if it's a vanilla Python list, and the parameters will be visible to both the optimizer and the GPU.

[mariusfacktor]

Great video! I like to find out what other PyTorch users think about, and these are some helpful "best practices"

[scottmiller]

I would run into the need to use gc and clear the cache when running pytorch lightning. If allowed to stop on its own, lightning would clean up after itself, but at least in preliminary runs, you'd see things were going south or want to stop iterating for some other reason, and breaking the iteration loop in lightning before it completed its iterations would leave all kinds of garbage (the still-loaded models, etc.) on the GPUs, as you could easily test by looking at nvidia-smi - the memory would still be in use, even though lightning was supposed to release it. I can't remember if I used your exactly your lines, but I ran something to clean up the GPUs. It convinced me (along with the relative inflexibility of lightning) to stop using lightning. I don't know if the lightning devs have cleaned this up in the intervening 4 years, but it's something to keep in mind.

[PritishMishra]

Amazing video! I didn't knew adding device='cuda' can make such difference!

[talkingbirb]

3rd tip: use torch.nn.ModuleList instead of list

[eduardopasseto]

Thank you very much, Edan. Hugs from Brazil!

[hassanhamidi]

Great video. Keep making videos like this, please

[haritoshpatel]

I am a PhD student focusing on machine learning and I can assure you these are amazing starter tips - sometimes I even forget occasionally the .eval one xD

[henryhsu]

5:24 essential issue! Thank you for sharing.

[stracci_]

Tip number one and already got my like 🤯

[pritamkarmokar]

Thank you for the video. It could have been better if the display were zoomed in a bit more.

[zenchiassassin]

Another tip: use torch.no_grad() or torch.inference_mode() at evaluation time

[fabiolaespinoza]

Thanks for the video ! For the 6th trick, do you know if replacing the model's instance frees up memory or does it accumulate? For example, if I first train a model using the instance name 'example_model' and then train another model with the same name, in terms of memory will I be accumulating the 2 models' space? Or just the last? Thanks!

[pullrequest]

The test time of the first case is obviously not correct since you do not synchronize before measuring time.

[wryltxw]

Is Pytorch as good as TF for high performance?

[wretchedmoose]

i dont get coding but this make me want to understand it

[death]

Why deploying in PyTorch is not the best option?

[MadlipzMarathi]

Use pytorch lighting you will avoid 70% of your mistakes

[jargolauda]

please use a bigger font in the video. code is taking oly 1/3 of the screen width, so there is empty space for bigger font, for us TV tubers and couch programmers