Multiprocessing in Python - Advanced Python 17 - Programming Tutorial

In this Python Advanced Tutorial, I will go into more detail about the multiprocessing module in Python. This video will cover:

- How to create and run multiple processes
- How to share data between processes
- Hot to use Locks to prevent race conditions
- How to use a Queue for process safe data exchanges
- How to use a Process Pool to manage multiple worker processes

~~~~~~~~~~~~~~ GREAT PLUGINS FOR YOUR CODE EDITOR ~~~~~~~~~~~~~~

📚 Get my FREE NumPy Handbook:

📓 Notebooks available on Patreon:

A written Tutorial can be found here:

You can find me here:

#Python

----------------------------------------------------------------------------------------------------------
* This is a sponsored link. By clicking on it you will not have any additional costs, instead you will support me and my project. Thank you so much for the support! 🙏

[SourabhDesaiBrief]

I have watched at least 7 videos before this, Thank you @Python Engineer, for your crucial part. You saved the day today!

[mugdhabapat]

Good basic tutorial for multiprocessing. I already am aware of multiprocessing concepts and also have learnt the application to Java. I was looking for the constructs in Python and this tutorial walked me through them systematically. Thanks.

[nitinagrawal]

For any one trying multithreading or multiprocess samples, either create time consuming task or add sleep() to make it time consuming. Otherwise benefit of these concepts will not be observed as output will possibly be sequential. One suggestion, this multithreading & multiprocessing concept is so captivating that without proper use case analysis can cause a disaster. And without considering locking & shared memory space, multithreading implementation causes a disaster in waiting.

[Damirgambit]

Thank you so much for good explanation. I have a question. How can I process 2d array with pool. And work with results outside a multiprocessing function

[surajshah]

Good tutorial thanks. @16:54 how would you write code where one process squares the number and a second process immediately follows to make the square number negative?

[ranjiths-bw]

for i in numbers:
numbers.value

iteration didn't work.
can you suggest other method for iteration.

[ludovikc]

Hey is there any way to make two processes share an object? Thanks

[ilhamwinar]

thank you, i will learn then but the explanation are completely great! hope u make a computer vision tutorial.

[ashinjohnson]

if the line print(q.get()) provides all the outputs of the process p1 & p2, then what will be the ordering in the q.get() like the process with finishes first will be the first element in q.get() or the it will depend the order of processes we defined like p1, p2...which is right?i like to know the from print(q.get()) which element belongs to which process for assing the output values to proceed....

[caiyu]

queue() from multipleprocess does not need to be lock-protected?

[kazaki]

I have an issue where my processes in a pool are Sleeping, any tips ?

[DennisAllard]

Good summary of multiprocessing with your data. The section of the end about pools is inadequate.

[UrsuIrinel]

Do you have any practical example for multiprocessing?

[jacobjonm]

Thanks for this. I tried running the cube function with two mapping approaches:

numbers =
Start = time.time(); result = list(map(cube, numbers)) ; time.time() - Start
Start = time.time(); result = pool.map(cube, numbers) ; time.time() - Start

Both of the mappings took 3.7 seconds to finish. But I could see in my system monitor the pool used all the 8 CPUs while the first mapping used only one CPU.
Is there any explanation for this?

[robv]

can you pass a class to multithreading? So it will create the class and run its __init__ method?

[FreeMarketSwine]

Is it worth using this on numpy or pandas data or are those libraries already maximally optimized?

[kyngcytro]

Quick question, does anybody have any idea on how to remove an item from the shared_array.

[aniruddh]

Hello! does multiprocessing create duplication? I am trying to process mails and the information is being stored in the database. However some of the information is being duplicated.
I don't know if it is because of the multiprocess.process() or if it is just the logic.

[undisclosedmusic]

Your audio could use a low-pass and some compression

[pawsdev]

Queue is awfull, cant manage big data flow, so will try shared memory. maybie it will be interesting to make video about multiprocessing.manager and concurrent.futures