jobqueue 0.04

jobqueue is a program for executing jobs in parallel to complete all jobs as fast as possible. The jobqueue project can be used to distribute a set of jobs to a specific number of processors and/or machines. New jobs are started as older jobs finish.

SYNTAX:

jobqueue [-e] [-n x] [-m list] [-v] [--version] [FILE ...]

jobqueue is a tool for executing lists of jobs on several processors or machines in parallel. jobqueue reads jobs (shell commands) from files. If no files are given, jobqueue reads jobs from stdin. Each job is executed in a shell environment (man 3 system).

-n x / --nodes=x, jobqueue keeps at most x jobs running in parallel. jobqueue issues new jobs as older jobs are finished.

-e / --execution-place, each job is executed by passing an execution place id as a parameter. The execution place defines a virtual execution place for the job, which can be used to determine a machine to execute the job. The place id is an integer from 1 to x (given with -n). If command "foo" is executed from a job list, jobqueue executes "foo x", where x is the execution place id.

-m list / --machine-list list, read contents of list file, and count each non-empty and non-comment line to be an execution place. Pass execution place for each executed job as a parameter. The difference to -e is that -e passes the execution place as an integer, but this option passes the execution place as a name for the job. Also, this option implies "-n x", where x is the number of names read from the file.

-v / --verbose, enter verbose mode. Print each command that is executed.

--version, print version number

EXAMPLE 1: A file named MACHINES contains a list of machines to process
jobs from a job file named JOBS. Each line in the JOBS file follows the
same pattern:
./myscript data0
./myscript data1
./myscript data2
...

MACHINES file contains 4 machines:
machine0
machine1
machine2
machine3

./myscript might do something like this:
#!/bin/bash

# This is the dataX parameter from JOBS file
data="$1"

# determine the machine that will execute this job
machine=$2

echo $machine $data
ssh $machine remotecommand $data

To execute jobs on the machines, issue:

jobqueue -m MACHINES JOBS

Execution will print something like this:
machine0 data0
machine1 data1
machine2 data2
machine1 data4
machine3 data3
machine0 data5
machine1 data7
machine2 data6
All jobs done (8)

EXAMPLE 2: run echo 5 times printing the execution place each time

for i in $(seq 5) ; do echo echo ; done |jobqueue -n2 -e

prints something like "1 2 1 2 1".

What's New in This Release:

· Machine list (-m) can now be used to specify the maximum number of simultaneous jobs for each machine.



Product's homepage