threading vs multiprocessing in python

A comparative look between threading and multiprocessing in python.

I will show activity plots of 4,8,16 threads vs 4,8,16 processes and discuss the differences between the two modules.

In summary: threads in python are concurrent and not parallel, so no two threads can execute at the same time. The way to get around this isto use the core module multiprocessing and spawn child python processes to each run work in parallel.

[sorcerer_of_supreme]

I can't imagine the effort and time you have invested for making this video.. Very informative

[EbonySeraphim]

Very nice full presentation. The short of it is that "Python" doesn't support parallel execution. For most programmers, when you talk about having multiple threads, the assumption is that those threads can and will execute in parallel. Unfortunately, Python was designed with single core CPU in mind so even though the idea of threads have existed for a while in computing, code wasn't likely to be run on a machine to do anything in parallel. It was just the operating system giving out small slices of time to execute one thread or another and it was perceived like both were happening at the same time -- very much like your graphs show.

Python, like most interpreted languages, cannot get over this problem because of the synchronization and locking needed to share access to data across threads so they inherently can only allow one "Python interpreted" thread to run at a time. Only library implementations in C can get around this under the hood by spawning real threads on Python's behalf to do work. Or this "multiprocess" approach, which creates a new process and an independent Python interpreter with entirely separate program state and memory. This approach isn't really a Python solution because any programming language can spawn a new OS process (provided a library is available to access fork() and exec*() system calls) and then the OS will execute that process in parallel on a multicore machine. But the thing about multiple processes is that it's harder and slower to share and synchronize data between processes than it is threads. It may not be an issue in some cases if not much synchronization is needed (the case if only an end result matters at the end of parallel work), but it can be a severe a limitation.

The last thing I'll say is that often times IO driven or IO heavy applications don't really need a performance boost of true parallel execution. The wait for IO (disk and network for example) are so slow compared to CPU execution that most threads would be waiting for IO anyways. With proper async-io setup (kqueue, select, epoll, IO completion ports) you can use a single thread to handle and dispatch thousands of IO requests and still be bottlenecked by IO. This is how/why people can still write "performance intensive" applications with interpreted languages and compete with a language like C or C++. Maximizing IO efficiency is simply something that sometimes C/C++ won't offer any benefit for so much "slower" languages appear to be just as fast.

[MrHvfan]

Thankyou Dave, i'm so glad youtube algo's bought your video to my daily feed. A really fascinating insight into thread and processes and the presentation style was perfect. Best wishes.

[ninjahkz]

this video is amazing, honestly one of the best I've ever seen, thank you from the bottom of my heart for dedicating so much time to creating it❤️

[calloq]

What an incredible video. I’ve just been blindly picking one or the other, not sure the differences between either one, but this makes everything so clear. I’m so glad I found it!!

[falwk]

Absolutely the best video on youtube describing how threading works in Python, with concise demonstrations and a well thought of script and presentation. 10/10, subscribed

[DChoi]

This is by far the most comprehensive and easily consumable video on any CS learning I've ever seen. Great job! Giving you a sub for sure.
Keep it up Dave!

[neelshah]

Excellent explaination about the most complicated questions that I have ever come across in an interview setting. Even though, this is an after math I am super glad to learn in with such a thought clarity. This is how you become fear-less!!! Thank you Dave ❤!

[aramshojaei]

This is the most comprehensive video I've ever seen on multithreading and multiprocessing. Great job!

[meme-getq]

This video was so informative, even for someone who is unfamiliar with the concept
You deserve a lot more recognition

[rampage_sl]

Brilliant work!! Best video on multithreading/processing I've seen in a while

[javedalam]

Brilliant representation of the concept. Thanks for all you effort.

[mehulaggarwal]

Literally the best video ive seen yet on this topic. Keep posting man!

[NONAME-eyqs]

This is hands down the most thorrow video on a topic. And youtube shows me this exactly 1 year after I desperately needed it.
Better late than never i guess.

[alexandrepv]

Very well explained :) I can see your number of subscribers growing at a steady pace mate. Keep it up! Good stuff!

[TON-vzpe]

Massively underrated video. Saved it in my library. Thank you sir.

[nj]

2 minutes into this I already understand it better than all other readings I did online. Nice!

[aeronesto]

This is the very best explanation of threading vs multiprocessing that I have ever seen. Well done!

[retrogamessocietybrasil]

One of the best lectures on multiprocessing and threading that I ever saw. Thanks for the guide and info, this will help me improve my own lectures on the subject

[yeahthisismyhandleyouknow]

Yay, it's so interesting to see a visual representation of something that I have been figuring out during my work for a few years with threading\multiproc.

When you understand it on instincts but not so visualized and vivid.