17 Running Multiple Commands in Parallel on a GNU Linux

[brunojardim]

Quando comecei a ver esse vídeo... estava jurando que você iria demonstrar com XARGS 😅

[thegripmaster]

There is actually one more way

parallel -j4 wget {} :::: img.list

That's 4 : to specify input is coming from file

[AxelWerner]

I want to run several external programms in parallel, while still be able to check and react on individual errors or return codes. So just by launching subcommands the job is not done. What then? Launching whole Shellscripts in parallel?

[LSSTS-smjx]

Very useful like always....thank you so much for sharing your knowledge and for the great job in your channel !!!

[wpribble]

Great info!!!! I learn something new from your channel all the time!

[NamasenITN]

The example of downloading from the internet is not optimal, as it conflicts with another bottleneck: bandwidth. So you have a limitation on number of cores or threads AS WELL AS the limits on the download bandwidth and speed of your ISP.

[krzysztofwaleska]

Great! I didn't know that. Had my own script for it. More videos like this would be great! Superb job!

[vanceg]

THANK YOU SO MUCH!!!! Running an Octave program, which doesn't multi thread, so I broke the program up into smaller pieces and ran them in parallel.

[jakemercer]

"-j" for jobs is the easiest way to remember I find, gcc has this for parallel compilation too :)

[zapy]

How do you get the optimum value for -j ? depending on the machine architecture

[SarahPoulin]

How would I use parallel for if statements? For example, I have certain conditions that need to be met, and then a file upload initiates to one platform using rclone. I want to also upload to another platform (not rclone) simultaneously. How would I do that?

[sakyb]

Thank for sharing this awesome tutorial..

[altaff]

How to put timeout against each process, example if child-process exceeds 60 sec, kill it

[jan_harald]

eh audio's slightly weird
but not a bad video

[berenscott]

Hey, I'm using dc3dd, and what I am looking to do is parallel wiping. So, I could for i in /dev/sd[a-z] then create a file with the list of targets /dev/sda /dev/sdb and so on. Then do something like parallel -j 4 dc3dd IF=/dev/null OF={} < list.txt

But, the problem is, how well will this display to the screen? The wget output seems to work fairly well.

[apdy]

This was exactly what I was looking for. I wanted to copy logs file from kubernetes pods but doing one at a time was taking a lot of time. this should really help.

[bestehefueralle]

Press F for Kris's shell that was assasinated by 14 wget jobs


We'll never forget you

[magicaryeh]

Gnu parallel is so nice they ask you to remember to cite them otherwise pay 10000 euros lol and then when you type that you will remember then they thank you in capital letters lol

[slaxblake]

what if instead of 1 line, there is a function we want to run in parallel, or almost a full script?