Threading vs Multiprocessing in Python - Advanced Python 15 - Programming Tutorial

Threading vs Multiprocessing in Python - Advanced Python 15 - Programming Tutorial

In this Python Advanced Tutorial, we will be learning about Threading and Multiprocessing in Python. With Threading and Multiprocessing you can run code in parallel and speed up your code. This Tutorial will cover:

- the difference between a Process and a Thread
- The advantages and disadvantages of both
- The Global interpreter lock (GIL)
- How and why Threads are limited by the GIL
- How to use the built-in threading and multiprocessing module to create and run multiple threads or processes.

~~~~~~~~~~~~~~ GREAT PLUGINS FOR YOUR CODE EDITOR ~~~~~~~~~~~~~~

📚 Get my FREE NumPy Handbook:

📓 Notebooks available on Patreon:

A written Tutorial can be found here:

You can find me here:

#Python

----------------------------------------------------------------------------------------------------------
* This is a sponsored link. By clicking on it you will not have any additional costs, instead you will support me and my project. Thank you so much for the support! 🙏

[ihgnmah]

Clear and detailed explanation. Many thanks for your work.

[donfeto]

Good job.

Remember the code will not run in windows without
if __name__ == '__main__':
from multiprocessing import freeze_support
freeze_support()

Here is the code

from multiprocessing import Process
from threading import Thread
import os
import time


def worker(name):
print(f" Processors {os.getpid()} is Running on {name}")
time.sleep(0.1)


if __name__ == '__main__':
from multiprocessing import freeze_support
freeze_support()
processors = []
num_processors = os.cpu_count()

if num_processors is None:
num_processors = 1

# create processes
for i in range(num_processors):
p = Process(target=worker, args=["Feto"])
processors.append(p)

# start each process
for p in processors:
p.start()

# Join the process to wait all the process to finsih then close the main thread that enable program to work.
for p in processors:
p.join()

[durandas]

Explaining was really good, I now somewhat feel like I understand it better than before..

[Dimineko]

Keep going like this. You are doing great.

[nintendo]

Hi! Just a bit more! How I can get the result value of a function executed in a Process? Or, maybe, I can write to a global list from the executed function on a process?

[thuthuy]

Can i ask some questions...
How many thread we should create ?
How many thread we can create ?

Please answer my questions.

[philipjohnson]

Have you investigated using CPU affinity with multiprocessing? I am using Windows 10 and am trying to get cpu affinity to work to set a specific core to run only a specific process.
I really enjoyed your videos.

[lakeguy]

I'm trying to wrap my head around multiprocessing. I have some code that seems ideal to be done in parallel. I have a large number of files (500) that I loop through 1 at a time. within each loop, a series of functions act on the file modifying it. at the end of each loop, a new file is created. (think of a factory where raw material goes in and a finished product comes out a the end of the process). using your code as a template, where would I loop through the 500 files? (substituting square_number for my functions/methods...) thank you!

[melellington]

I am using a MacOS, with a 6 core Intel i5, so I got 6 threads.
The code in the video would not work in my PyCharm IDE
using Python 3.9. It gave me MANY errors. I fixed them
using "if __name__ == '__main__'" and a def main():

from multiprocessing import Process
import os
import time

def square_numbers():
for i in range(100):
i + i
time.sleep(0.1)

def def main():
processes = []
num_processes = os.cpu_count()

for i in range(num_processes):
p =
processes.append(p)

for p in processes:
p.start()

for p in processes:
p.join()

print('end main')

if __name__ == '__main__':
main()

[hackdesigner]

Hey, great video.
I think it's important to say that despite the interfaces are so similar in Python, from the OS standpoint these two are immensely different things.
Coming from dotnet, where the API is drastically different which helps to grasp/guess the distinction too.
Also, really interesting, how the join() on process works in Windows, is it WinApi's WaitForSingleObject() or something alike? Will read about it.

[marcoantonioarmentaarmenta]

I don't understand if the p.start() will call the function square_numbers 4 times (number of your cpus) or partition the for loop inside square_numbers into 4 pieces and run each one on a cpu.

[animlwild]

Gives error when at the end of execution says attempt to start a new process before current process finished bootstrapping

[emils-j.]

Question:

Why not

for p in processes:
p.start()
p.join()

?

[yaminiprabakaran]

if __name__ == "__main__": is needed in the program for windows
its weird..In my windows pc, when I run the program, end main at the end of the program is getting printed 9 times..could you plz check and comment?
for me, num_processes is 8(I think it is 8 logical cores)

from multiprocessing import Process
import os
import time

def square_numbers():
for i in range(100):
i * i
time.sleep(0.1)

if __name__ == "__main__":
processes = []
num_processes = os.cpu_count()

#create processes
for i in range(num_processes):
p =
processes.append(p)

#start
for p in processes:
p.start()
#join
for p in processes:
p.join()

print('end main')

[utayasurian]

How to avoid broken process pool in multiprocessing?

[leonardopetribu]

Excellent video! Btw, which IDE and color theme are those? Thanks!

[dk]

That you very much to explain why GIL is used. It was very helpful

[nischithnishanimath]

Getting "PermissionError: [WinError 5] Access is denied"

[tiktoktiktik]

How can there be a race condition if only one thread executes at a time?

[shokhrukhabduahadov]

After watching this video, i got confused and my watching other videos goes to trash along with this one