Unlocking your CPU cores in Python (multiprocessing)

How to use all your CPU cores in Python?

Due to the Global Interpreter Lock (GIL) in Python, threads don't really get much use of your CPU cores. Instead, use multiprocessing! Process pools are beginner-friendly but also quite performant in many situations. Don't fall into some of the many traps of multiprocessing though, this video will guide you though it.

SUPPORT ME ⭐
---------------------------------------------------

Top patrons and donors: Jameson, Laura M, Vahnekie, Dragos C, Matt R, Casey G, Johan A, John Martin, Jason F, Mutual Information, Neel R

BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------

MUSIC
---------------------------------------------------
It's a sine wave. You can't copyright a pure sine wave right?

[tdug]

It's also worth noting that smaller chunk sizes may be better for unpredictably distributed job times, as one runner may randomly grab many expensive jobs, and lock the pool when the rest of the processes finish.

Great video, as always!

[ArtML]

\o/ Yay! Long waited multiprocessing video! Always appreciate the humor in intros! :D
Thanks a lot, I am on a path of making parallelization / multiprocessing to become a second nature in my coding - these videos help greatly!
More topic suggestions:
- Simple speed-ups using GPUs
- Panda speedup by Dask - unlocking multiple cores
- Numba, JAX and the overview of JIT compilers
- Cython, and the most convenient (easy-to-use) wrappers for C++ implementations
- All about Pickling, best practices/fastest ways to write picklers for novel objects

[unusedTV]

Your video is about two years late for me!
I was working on a heat transfer simulation in Python where we had to compare hundreds of different input configurations. I knew about the GIL and multiprocessing in general outside of Python, but had to figure out myself how to get it to work. Eventually I settled on a multiprocessing pool and it worked wonders, because now we could run 32 simulations in parallel (Threadripper 1950x).

Quick caveat that I don't hear you mention: a lot of processors have hyperthreading/SMT (intel/amd respectively), showing double the amount of cores in the task manager. In our case we found that spawning a process for each physical core provided better results than using all logical cores.

[SpeedingFlare]

That pool thing is so cool. I like that it spawns as many processes as there are cores available. I wish my work had more CPU bound problems

[lawrencedoliveiro]

5:07 Threading is also useful for turning blocking operations into nonblocking ones. For example, asyncio provides nonblocking calls for reading and writing sockets, but not for the initial socket connection. Simple solution: push that part onto a separate thread.

[michaellin]

The funny thing is, adding random noise is actually a useful thing to do. It's called dithering, and is used nearly everywhere in signal processing.

[jakemuff]

Great video! Maybe some more "real world" examples would be useful. Knowing that my code *could* be parallelized and actually parallelizing the code are two very different things. I've found that knowledge of multithreading in python does not translate to automatic code speed up. And of course no two problems are the same.

[jesuscobos]

Love your videos. I usually watch all of them just for fun but this has enabled me to speed up a very heavy optimization for my science stuff. Ty for your dedication. I can ensure that it has real world implications :)

[OutlawJackC]

Your explanation of the GIL makes so much more sense than other people :)

[lawrencedoliveiro]

9:27 Actually, there is a faster way of sharing data between processes than sending picklable objects over pipes, and that is to use shared memory. Support for this is built into the multiprocessing module. However, you cannot put regular Python objects into shared memory: you have to use objects defined by the ctypes module. These correspond to types defined in C (as the name suggests): primitive types like int or float, also array and struct types are allowed. But avoid absolute pointers.

[zemonsim]

This video was so helpful ! I recently converted my mass encryption script to use multiprocessing. To encrypt my dataset of 450 Mb of images, it went from an estimated 11 hours to just 10 minutes, doing the work at around 750 Kb per second.

[michaelwang]

The clearest explanation on this topic that I have ever seen! Really nice!! Thanks for sharing!

[etopowertwon]

Multiprocessing helped me a lot recently. I had a script that periodically loads lots tons of small XML from netshare, process them and save locally, single thread ran in 30 seconds, multiprocessed ran in about 6 seconds.

[daniilorekhov]

Would love to see a video on managing shared memory in multiprocessing scenarios

[zlorf_youtube]

Me learns a new python thing... Starts using it in every fucking uncessary place. Feels good.

Really good thing to talk about typical pitfalls.

[knut-olaihelgesen]

You are actually the best at advanced python videos! Love them so much

[unotoli]

So well explained. One nice2have thing - quick tip on how to debug (see summary of time2process) most cpu-intensive tasks (functions, like wav transformation in this case).

[superzcreationz]

This is what I was searching very very useful video...😊...that's why I subscribed you 😊

[eldarmammadov]

liked the way to speak about all three modules asynchio, threading, multiprocessingin one vidoe

[nocturnomedieval]

This is so good and clear. A must share. BTW, how these techniques relate to the case when you are using numba with option parallel=True?