Difference: WmsxReference (1 vs. 2)

-v

Specifies that you want debug output on the console.

myworkdir

Location of the working directory.

-f

Check if the provider is running.

-k

Kill provider.

-rememberafs

Will ask for your AFS password and renew AFS tokens

-forgetafs

Forget the password.

-remembergrid

Will ask for your Grid password and renew the

-vo yourvo

Set the VO for all future job submissions to yourvo.

Traditional Submission of Single Jobs with Traditional JDL Files

Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is:

-j

Name of the JDL file.

-o

If the JDL file has JobType set to "Interactive",

-r

When the job is done, results are retrieved and stored in the

Automated Mass Submission of Jobs via ArgList Files

-a args.file

Submit many jobs with the ArgList file args.file.

-name runname

Give the name runname to this execution run (optional).

-n nmaxjobs

Set the maximal number of concurrently running jobs

The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl, which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed.

WMSX JDL files.

For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job. The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:

Archive

Name of the program archive file. This is the name of the

ProgramDir

Name of the root directory inside the program archive

Executable

Name of the executable to run inside the ProgramDir.

OutputDirectory

The name of the directory under ProgramDir, where

JobType

If this variable is set to "Interactive", the jobs will

Software

List of software that must be present (executable) on the

Requirements

Extra queue requirements, like in a traditional JDL file.

If the JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).

In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension, if JDL file is not present).

PreExec and PostExec Scripts

In most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be called COMMAND_preexec and COMMAND_postexec. They must be executable. They will be run directly before submission and after job output is retrieved, respectively.

PreExec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments.

PostExec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framwork), and with all the given arguments from ArgList as following arguments.

The retrieved outputs of your job will always be in the tarball OutputDirectory.tar.gz under the job output directory, where OutputDirectory is what you specified in the JDL file (out by default).

If PostExec returns with 0 nothing further happens. If PostExec returns with 1, the user script COMMAND_chain is called, which can be used to launch further jobs.

Job Chaining

The running time of Grid jobs is limited. This is unavoidable for efficient control on resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state should be dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The script COMMAND_chain is the tool to lauch further jobs, when needed.

If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.

The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.)

WMSX

Basic usage

There are two program you will be using: Provider and Requestor.

Provider

is the background process which does the actual work. There can be only one provider runnung on a machine per user. Start with

./wmsx-provider -v /tmp/myworkdir

where:

-v

Specified you want debug output on the console. Leave this out once you feel comfortable.

/tmp/myworkdir

Location of the work directory. This is where all your output goes. You may want to set this to a location inside your home directory.

Requestor

is an application to submit commands to the system for execution. Start with:

./wmsx-requestor -option

A complete list of options is available with -h. Some options are:

-f

Check if the provider is running

-k

kill provider

-rememberafs

Will ask for your AFS password and renew AFS tokens until you use

-forgetafs

to forget the password

-remembergrid

Will ask for yout Grid password and renew the Grid tokens until there are no more managed jobs

-n

set the maximum number of concurrently running jobs

-vo

set the VO for all future job submissions

JDL Files

Can be sumbitted and managed through the framework. A sample usage is:

./wmsx-requestor -j example.jdl -o /tmp/StdOut -r /tmp/resultDir

Where the options are:

-j

name of the JDL file

-o

If the JDL file has "JobType" set to "Interactive", then StdOut / StdError will be retrieved while the Job is running and stored in the given filename

-r

When the job is done, results are retrieved and stored in the given directory.

ArgList Files

Can specify multiple jobs to create one "larger" jpb.

ArgList Format:

COMMAND parameters

./wmsx-requestor options:

-a

args.file

-name

name of this execution. Is used as directory to distinguish different runs with the same commands.

Custom JDL files.

for Each command, there may be a file command.jdl, with custom JDL contents. sup ported are:

Archive

name of the archive, defaults to COMMAND.tar.gz (must be tar-gz format!)

ProgramDir

name of the program dir inside the tar file, defaults to COMMAND

Executable

name of the executable to run inside the program dir, defaults to COMMAND

OutputDirectory

Which directory to retrieve, defaults to "out"

JobType

If set to "Interactive", the jobs will be run as interactive

Software

List of software that must be present (executable) on the target machine. The special key "AFS" checks for AFS presence

Requirements

Extra requirements, as specified by the original JDL spec.

Pre/PostExec

must be calles COMMAND_preexec and COMMAND_postexec. MUST be executable. Will be run directly before submission and after job output is retrieved.

Preexec is called with the name of the command and its full directory as first argument, and with all the given args as following arguments.

Postexec is called with the name of the command and its full directory as first argument, , the name of the output directory as second argument, and with all the given args as following arguments.

The output of your job will always be in a file "out.tar.gz" in the output directory.

If postexec return with 0 nothing further happens. If postexec returns with 1, COMMAND_chain is called.

Chaining

The _chain command's output is interpreted just as a line from the args file. It can therefore be used to start further jobs.