Easily Parallelize Commands in Linux (Multiprocessing)

A while ago I created a script in Python that would

  • automatically find out the number of vCPUs on a computer.
  • read a file which is supposed to list commands on each line.
  • execute each command on its own process, and running only the number of processes as there are vCPUs.
    • This is important as we want the speed of parallelizaton but not to overload the computer, causing lots of wasted overhead in the CPU swapping in and out of processes.

Do not use these scripts to execute a series of commands that are dependent on each other and need to be executed in a certain order.

The script was just 24 lines long and worked well. However, if you are in Linux, you can achieve the same result in just two lines using xargs.

NUM_PROCS=`cat /proc/cpuinfo | awk '/^processor/{print $3}'| wc -l`
< $@ xargs -d '\n' -P $NUM_PROCS -I {} /bin/bash -c "{}"

This first line figures out how many vCPUs you have in a way that should hopefully work on any computer, no matter how your processors are indexed etc.

With regards to the second line:

  • the &lt; $@ is passing the first argument (which should be the path to the file of commands) to standard input.
  • The -d '\n' is specifying that the newline is being used for the delimiter between arguments to xargs.
  • The -P $NUM_PROCS tells xargs to maintain a number of processes equal to the number of vCPUs we have (you can manually set $NUM_PROCS if you desired). * -I {} is telling xargs to substitute {} in the bit coming up with a line we read from the file (an argument passed to xargs).
  • The /bin/bash -c "{}" is telling xargs to use bash to execute the line which is passed in as a string.

Usage

  • Copy and paste the two line script into a file called multiprocess in your $PATH.
  • Create a text file with all the commands you want to run in parallel. E.g.
sudo virsh snapshot-create server1.programster.org  
sudo virsh snapshot-create server2.programster.org  
sudo virsh snapshot-create server3.programster.org  
sudo virsh snapshot-create server4.programster.org  
...
  • Now execute multiprocess /path/to/commands/file to process those commands in parallel.

References

Author

Programster

Stuart is a software developer with a passion for Linux and open source projects.

comments powered by Disqus