jobqueue is a program for executing jobs in parallel to complete all jobs as fast as possible. The jobqueue project can be used to distribute a set of jobs to a specific number of processors and/or machines. New jobs are started as older jobs finish.
jobqueue [-e] [-n x] [-m list] [-v] [--version] [FILE ...]
jobqueue is a tool for executing lists of jobs on several processors or machines in parallel. jobqueue reads jobs (shell commands) from files. If no files are given, jobqueue reads jobs from stdin. Each job is executed in a shell environment (man 3 system).
-n x / --nodes=x, jobqueue keeps at most x jobs running in parallel. jobqueue issues new jobs as older jobs are finished.
-e / --execution-place, each job is executed by passing an execution place id as a parameter. The execution place defines a virtual execution place for the job, which can be used to determine a machine to execute the job. The place id is an integer from 1 to x (given with -n). If command "foo" is executed from a job list, jobqueue executes "foo x", where x is the execution place id.
-m list / --machine-list list, read contents of list file, and count each non-empty and non-comment line to be an execution place. Pass execution place for each executed job as a parameter. The difference to -e is that -e passes the execution place as an integer, but this option passes the execution place as a name for the job. Also, this option implies "-n x", where x is the number of names read from the file.
-v / --verbose, enter verbose mode. Print each command that is executed.
--version, print version number
EXAMPLE 1: A file named MACHINES contains a list of machines to process
jobs from a job file named JOBS. Each line in the JOBS file follows the
MACHINES file contains 4 machines:
./myscript might do something like this:
# This is the dataX parameter from JOBS file
# determine the machine that will execute this job
echo $machine $data
ssh $machine remotecommand $data
To execute jobs on the machines, issue:
jobqueue -m MACHINES JOBS
Execution will print something like this:
All jobs done (8)
EXAMPLE 2: run echo 5 times printing the execution place each time
for i in $(seq 5) ; do echo echo ; done |jobqueue -n2 -e
prints something like "1 2 1 2 1".
What's New in This Release:
· Machine list (-m) can now be used to specify the maximum number of simultaneous jobs for each machine.