Difference: WmsxExample (3 vs. 4)

Revision 42007-11-29 - AndrasLaszlo

Line: 1 to 1

META TOPICPARENT	name="WmsxSoftware"

WMSX examples

Changed:

<
<

Simple job

>
>

Simple job submission with traditional JDL files

Job submission with edg-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.

Line: 47 to 47

Mass submission of independent jobs: simple parameter scan

Added:

>
>

WMSX is a very convenient tool for mass submission of independent jobs. Possible application: parameter scan studies. It automatically manages output retrieval issues and the limiting of the number of concurrently running jobs. For this purpose, the jobs are described with a slightly extended JDL language, which is preprocessed by WMSX. If JobType = "Interactive" is set, the file containing StdOut / StdErr is updated on the fly, so you can see what your job is actually doing. A pre-execution script helps to prepare inputs, and a post-execution script helps to process outputs of the job.

You can find a prepared example for this task here: . To submit the example job, type the following commands.

Get the tarball:

> tar  -xzf  sample.tar.gz
> rm  -f  sample.tar.gz
> cd  sample

Log onto the Grid:

> voms-proxy-init  -voms your_vo
# If your jobs are expected to be long, also get long term authentication:
> myproxy-init  -n

Submit the jobs:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
# If your jobs are expected to be long, also make the WMSX remember your grid password:
> wmsx-requestor.sh  -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
> wmsx-requestor.sh  -rememberafs
# If you want to submit many-many jobs, limit the number of concurrently running jobs e.g. to 100:
> wmsx-requestor.sh  -n 100
> wmsx-requestor.sh  -a arg.list

The file arg.list contains a list of jobs (together with its parameters) to be started. Your results shall be retrieved under the directory workdir/out. If your job was interactive (JobType = "Interactive" was set in the JDL file), the file containting StdOut / StdErr shall be updated on the fly.

If you are finished, you can stop WMSX, and destroy your Grid authentications:

> wmsx-requestor.sh  -k
# If myproxy-init was used, i.e. for long term jobs:
> myproxy-destroy
> voms-proxy-destroy

Mass submission of chained jobs: parameter scan with chained jobs

Added:

>
>

The mass submission has an important feature: each job can decide whether a launch of a further job is needed. This decision is done by the post-execution script: if it returns with exitcode 1, the chain script is invoked. The lines of the standard output of the chain script is interpreted by WMSX as lines in the arg.list file (see previous example), so new jobs are launched. In this way, jobs can be chained (and the chains can also fork, as a job can also launch multiple jobs). In this way, a parameter study with long term calculations can be performed. The long term calculations are preformed by the job chains. A good example is numerical solution of partial differential equations: multiple chains are launched (possibly with different initial conditions), and each chain is imitating a long term job (the solution of the partial differential equation with a given initial condition), split up into subsequent shorter term jobs.

You can find a prepared example for this task here: . To submit the example job, type the following commands. (Similar to the previous example.)

Get the tarball:

> tar  -xzf  chainsample.tar.gz
> rm  -f  chainsample.tar.gz
> cd  chainsample

Log onto the Grid:

> voms-proxy-init  -voms your_vo
# If your jobs are expected to be long, also get long term authentication:
> myproxy-init  -n

Added:

>
>

Submit the jobs:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
# If your jobs are expected to be long, also make the WMSX remember your grid password:
> wmsx-requestor.sh  -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
> wmsx-requestor.sh  -rememberafs
# If you want to submit many-many jobs, limit the number of concurrently running jobs e.g. to 100:
> wmsx-requestor.sh  -n 100
> wmsx-requestor.sh  -a arg.list

Added:

>
> The file arg.list contains a list of jobs (together with its parameters) to be started. Your results shall be retrieved under the directory workdir/out. If your job was interactive (JobType = "Interactive" was set in the JDL file), the file containting StdOut / StdErr shall be updated on the fly.

Changed:

<
< -- AndrasLaszlo - 28 Nov 2007

>
>

If you are finished, you can stop WMSX, and destroy your Grid authentications:

> wmsx-requestor.sh  -k
# If myproxy-init was used, i.e. for long term jobs:
> myproxy-destroy
> voms-proxy-destroy

View topic | History: r18 < r17 < r16 < r15 | More topic actions...

Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback