Multiprocessing in Python

In this video we learn about multiprocessing in Python.

◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾

📚 Programming Books & Merch 📚

🌐 Social Media & Contact 🌐

[AecapA]

TUTORIAL IS WRONG!
What is wrong:
1. When creating the sub-process, all the names (variables) from the original process are being copied, but they are NOT shared with the original process. When you append to a list in the sub-process, list in the original process stays empty.
This style of creating sub-processes in the video is very similar to C++ and it's <thread> library. But in C++ to get results of calculations you pass to a function a pointer to a specific memory address to save results to, which is NOT the case for Python!
Basically, results_a in sub-process is not an instance of results_a in original process.
If you don't believe me, go ahead and try printing the results_a list after the calculations in the sub-process! It will be empty.

2. After starting the sub-process p1, you must JOIN it with the p1.join() command. This command waits for p1 sub-process to finish.
In the tutorial, program just starts sub-processes, but they may or may not be finished by the time the original process ends (in the case of finishing before sub-processes finish, it will throw an error I believe). And the time that you have measured - it is the time that it took to CREATE all the sub-processes, but not to finish them.

3. As many of commenters have said, temp_a = results_a does NOT make a copy of the list! After this line of code temp_a and results_a will "point" to the same list, and changing results_a will change temp_a. To check this, try print(results_a is temp_a) - this will print True! Therefor print(results_a == temp_a) will always be True, even if calculations in sub-process is wrong (which they are).

I have watched some videos from this channel before (for example a video "Progress Bars in Python Terminal" was very useful), but some videos are to short to cover all the nuances (which is very important in my opinion) and some are just catchpenny, botched and harmful for the viewer (like this one)! I really hope it is not intentional.
I wish the author good luck and be more responsible when creating videos.

[ThomasRStevenson]

Don't you need a p1.join() to wait the all of the processes to finish? Isn't the time you are recording just the time for the processes to start, but not to finsih (without the p1.join())?

[alierencelik]

There is an issue with list comparisons. You dont copy elements of the list but rather you generate another variable pointing at the list. No matter what calculations are made, they will always be equal. Thanks for the video btw.

[__wouks__]

I am not sure if I am correct on this but you declare your results_a, results_b and results_c lists and then append your calculations to these lists. Then you basically point to the same lists with your temp_a, temp_b and temp_c variables and append again to result_a, result_b and result_c. The thing is as I understand it temp_a and result_a are referring to the exact same list and therefor you are appending to both, which means temp_a and result_a will always be the same. If you use the "is" keyword instead of "==" it will also show true. They will also have the same id.

[ItsCloudHub]

After starting processes, I think we have to check if all those processes are completed, using .join(), I guess

where as you re directly printing end time immediately after starting process, so not sure if overall logic is correct

[InspektorDreyfus]

There should be a waiting point until the three parallel processes are finished before taking the end time stamp.

[davidpaipa]

I discovered this channel a couple of weeks ago and I love it !

[jairoantoniomeloflorez]

Men, thanks a lot for this mini-lesson :) It was very useful to overcome an issue that I had with a current project.

[richard_noblockhit]

9:35 Uhh, idk if im stupid, but those lists are immutable, you would need to deepcopy them to actually compare them and be sure that they end up with the same result. I trust you tho, this is supposed to work. otherwise well done!

[boazfortuijn]

I watched like two video's on this, but this one was by far the clearest. Thanks!!

[fxtudnc]

Hello! I use Threading with Selenium, download 2 profiles from the temple! through threadings, I pass the link and --profile-directory to args
. The problem is that on one thread selenium follows the link, on the other the browser opens and that's it... does not click on the and if I remove the path to the profiles and their directories, everything works fine! thank you in advance!!!

[muthurubant]

You are 😳😳😳.. I don't have words to appreciate you bro.. 🔥🔥🔥🔥

[IAGIC]

This needs an edit or even a full reupload. It doesn't make sense to say that there is a 7x speedup when you are only using 3 processes. At 100% efficiency you would finish 3 times faster, which even that won't really happen. You're just measuring the time it takes to launch the processes.

[suryaravikumar]

yeah buddy, thanks this is what i expected from you

[Gametime]

im deploying a python API file on amazon ec2 where one function in it utilizes multiprocessing to parallelize some data processing - assigned 2 CPU cores1
Now if this api is deployed and if the api receives 100 concurrent calls will api fail because it exceeded the processor capacity or any other cause to fail???

[lightgrid]

You were correct the first time. It's pronounced processes not processeeez

[ottomarjo]

This video finally helped me understand how multiprocessing works. Thank you!

[anonlegion]

how do you differentiate multi-threading vs multiprocessing? like which use case is good for multi-threading vs multiprocessing? which is faster?
Also, why aren't you developing on a linux distro?

[walnut]

Tysm for this video! It’s super useful for developing my physics sandbox in python. It’s found a great use!
Incredible video overall!

[XPxpxpXP]

But when I print out the a, b, and c after the mp calculation, they are empty lists.