Difference: WmsxExample (1 vs. 18)

Revision 182010-01-27 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 55 to 55
 # ... the above stuff comes here, but put ' ; \' instead of new line, and continue in next line ... (for tcsh).
Added:
>
>
If your home is on AFS, it is also advised to move also your working directory under /tmp/$USER, together with all the necessary files and directories, described in the following sections. To prevent that the regular cleanup cron job deletes some of your files from /tmp/$USER when they are too old, run the command
find  /tmp/$USER  -print  -exec touch {} \;
after unpacking the below tarballs into /tmp/$USER.
 

Simple job submission with traditional JDL files

Job submission with glite-wms-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.

Revision 172009-07-10 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 186 to 186
 > glite-voms-proxy-destroy
Changed:
<
<
-- AndrasLaszlo - 04 Nov 2008
>
>
-- AndrasLaszlo - 10 Jul 2009
 
META FILEATTACHMENT attr="" autoattached="1" comment="Interactive example" date="1208360461" name="interactive.tar.gz" path="interactive.tar.gz" size="529" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Simplejob example" date="1208360425" name="simplejob.tar.gz" path="simplejob.tar.gz" size="501" user="Main.AndrasLaszlo" version="1"

Revision 162009-07-10 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 93 to 93
 If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Deleted:
<
<
> killall java # <- This is necessary due to a WMSX bug.
 # If myproxy-init was used, i.e. for long term jobs: > myproxy-destroy -d
> glite-voms-proxy-destroy
Line: 139 to 138
 If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Deleted:
<
<
> killall java # <- This is necessary due to a WMSX bug.
 # If myproxy-init was used, i.e. for long term jobs: > myproxy-destroy -d
> glite-voms-proxy-destroy
Line: 183 to 181
 If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Deleted:
<
<
> killall java # <- This is necessary due to a WMSX bug.
 # If myproxy-init was used, i.e. for long term jobs: > myproxy-destroy -d
> glite-voms-proxy-destroy

Revision 152009-07-03 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Changed:
<
<
Note: There is an installed copy of WMSX software on the KFKI AFS. To activate it, follow the instructions of that homepage.
>
>
Note1: There is an installed copy of WMSX software on the KFKI AFS. To activate it, follow the instructions of that homepage.
 
Changed:
<
<
Note: If your home directory is located on AFS, it is advised to source the following shell script before logging on to the Grid:
>
>
Note2: The WMSX can work without the User Interface environment. For this, you only need an installed copy of WMSX on your computer, the usual ~/.globus/usercert.pem, ~/.globus/userkey.pem files and the ~/.glite directory as on any User Interface machine, plus you need to provide the ~/.globus/certificates directory, which can be obtained from any User Interface machine from /etc/grid-security/certificates. These files and directories can also be relocated by the X509_USER_CERT, X509_USER_KEY, GLITE_USER_HOME and X509_CERT_DIR environmental variables.

Note3: If your home directory is located on AFS, it is advised to source the following shell script before logging on to the Grid:

 
## Make new .globus directory and export it:
chmod  -R  755  /tmp/$USER/globus  >&/dev/null
Line: 12 to 14
 mkdir -p /tmp/$USER/globus cp $HOME/.globus/usercert.pem /tmp/$USER/globus/ cp $HOME/.globus/userkey.pem /tmp/$USER/globus/
Added:
>
>
cp $HOME/.globus/certificates /tmp/$USER/globus/
 export X509_USER_CER=/tmp/$USER/globus/usercert.pem export X509_USER_KEY=/tmp/$USER/globus/userkey.pem
Added:
>
>
export X509_CERT_DIR=/tmp/$USER/globus/certificates
 ## Make new .glite directory and export it: chmod -R 755 /tmp/$USER/glite >&/dev/null rm -rf /tmp/$USER/glite
Line: 27 to 31
 mkdir -p /tmp/$USER/globus cp $HOME/.globus/usercert.pem /tmp/$USER/globus/ cp $HOME/.globus/userkey.pem /tmp/$USER/globus/
Added:
>
>
cp $HOME/.globus/certificates /tmp/$USER/globus/
 setenv X509_USER_CER /tmp/$USER/globus/usercert.pem setenv X509_USER_KEY /tmp/$USER/globus/userkey.pem
Added:
>
>
setenv X509_CERT_DIR /tmp/$USER/globus/certificates
 ## Make new .glite directory and export it: chmod -R 755 /tmp/$USER/glite >&/dev/null rm -rf /tmp/$USER/glite mkdir -p /tmp/$USER/glite cp -rd $HOME/.glite/* /tmp/$USER/glite/ setenv GLITE_USER_HOME /tmp/$USER/glite
Changed:
<
<
(for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory).
>
>
(for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory). You could also put these inside a shell function or alias in your .bashrc or .tcshrc, like:
### Function to export the grid stuff in $HOME to the /tmp/$USER:
exportlocal()
{
    # ... the above stuff comes here ...
}
(for bash), or
### Alias to export the grid stuff in $HOME to the /tmp/$USER:
alias  exportlocal  \
# ... the above stuff comes here, but put ' ; \' instead of new line, and continue in next line ...
(for tcsh).
 

Simple job submission with traditional JDL files

Revision 142008-11-21 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 20 to 20
 mkdir -p /tmp/$USER/glite cp -rd $HOME/.glite/* /tmp/$USER/glite/ export GLITE_USER_HOME=/tmp/$USER/glite
Changed:
<
<
(for bash), or
>
>
(for bash), or
 ## Make new .globus directory and export it: chmod -R 755 /tmp/$USER/globus >&/dev/null rm -rf /tmp/$USER/globus
Line: 37 to 35
 mkdir -p /tmp/$USER/glite cp -rd $HOME/.glite/* /tmp/$USER/glite/ setenv GLITE_USER_HOME /tmp/$USER/glite
Changed:
<
<
(for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory).
>
>
(for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory).
 

Simple job submission with traditional JDL files

Line: 56 to 52
  Log onto the Grid:
Changed:
<
<
> voms-proxy-init -voms your_vo
>
>
> glite-voms-proxy-init -voms your_vo
 # If your job is expected to be long, also get long term authentication: > myproxy-init -d -n
Line: 82 to 78
 > killall java # <- This is necessary due to a WMSX bug.
# If myproxy-init was used, i.e. for long term jobs: > myproxy-destroy -d
Changed:
<
<
> voms-proxy-destroy
>
>
> glite-voms-proxy-destroy
 

An educative example for simple interactive job submission may be found here: interactive.tar.gz. This works in a similar way as the simplejob.tar.gz example.

Line: 102 to 98
  Log onto the Grid:
Changed:
<
<
> voms-proxy-init -voms your_vo
>
>
> glite-voms-proxy-init -voms your_vo
 # If your jobs are expected to be long, also get long term authentication: > myproxy-init -d -n
Line: 128 to 124
 > killall java # <- This is necessary due to a WMSX bug.
# If myproxy-init was used, i.e. for long term jobs: > myproxy-destroy -d
Changed:
<
<
> voms-proxy-destroy
>
>
> glite-voms-proxy-destroy
 

Mass submission of chained jobs: parameter scan with long calculations

Line: 146 to 142
  Log onto the Grid:
Changed:
<
<
> voms-proxy-init -voms your_vo
>
>
> glite-voms-proxy-init -voms your_vo
 # If your jobs are expected to be long, also get long term authentication: > myproxy-init -d -n
Line: 172 to 168
 > killall java # <- This is necessary due to a WMSX bug.
# If myproxy-init was used, i.e. for long term jobs: > myproxy-destroy -d
Changed:
<
<
> voms-proxy-destroy
>
>
> glite-voms-proxy-destroy
 
Deleted:
<
<
 -- AndrasLaszlo - 04 Nov 2008
Deleted:
<
<
 
META FILEATTACHMENT attr="" autoattached="1" comment="Interactive example" date="1208360461" name="interactive.tar.gz" path="interactive.tar.gz" size="529" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Simplejob example" date="1208360425" name="simplejob.tar.gz" path="simplejob.tar.gz" size="501" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Sample example" date="1208360496" name="sample.tar.gz" path="sample.tar.gz" size="1340" user="Main.AndrasLaszlo" version="1"

Revision 132008-11-13 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Added:
>
>
Note: There is an installed copy of WMSX software on the KFKI AFS. To activate it, follow the instructions of that homepage.
 Note: If your home directory is located on AFS, it is advised to source the following shell script before logging on to the Grid:
## Make new .globus directory and export it:
Line: 39 to 41
 (for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory).
Deleted:
<
<
Note: There is an installed copy of WMSX software on the KFKI AFS. Activate it as indicated on the KFKI AFS homepage.
 

Simple job submission with traditional JDL files

Job submission with glite-wms-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.

Revision 122008-11-13 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 39 to 39
 (for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory).
Added:
>
>
Note: There is an installed copy of WMSX software on the KFKI AFS. Activate it as indicated on the KFKI AFS homepage.
 

Simple job submission with traditional JDL files

Job submission with glite-wms-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.

Revision 112008-11-07 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Note: If your home directory is located on AFS, it is advised to source the following shell script before logging on to the Grid:

## Make new .globus directory and export it:
Changed:
<
<
if -d /tmp/$USER/globus ; then chmod -R 755 /tmp/$USER/globus ; fi
>
>
chmod -R 755 /tmp/$USER/globus >&/dev/null
 rm -rf /tmp/$USER/globus mkdir -p /tmp/$USER/globus cp $HOME/.globus/usercert.pem /tmp/$USER/globus/
Line: 13 to 13
 export X509_USER_CER=/tmp/$USER/globus/usercert.pem export X509_USER_KEY=/tmp/$USER/globus/userkey.pem ## Make new .glite directory and export it:
Changed:
<
<
if -d /tmp/$USER/glite ; then chmod -R 755 /tmp/$USER/glite ; fi
>
>
chmod -R 755 /tmp/$USER/glite >&/dev/null
 rm -rf /tmp/$USER/glite mkdir -p /tmp/$USER/glite
Changed:
<
<
cp -rd $HOME/.glite/vomses /tmp/$USER/glite/
>
>
cp -rd $HOME/.glite/* /tmp/$USER/glite/
 export GLITE_USER_HOME=/tmp/$USER/glite (for bash), or
## Make new .globus directory and export it:
Changed:
<
<
if ( -d /tmp/$USER/globus ) then chmod -R 755 /tmp/$USER/globus endif
>
>
chmod -R 755 /tmp/$USER/globus >&/dev/null
 rm -rf /tmp/$USER/globus mkdir -p /tmp/$USER/globus cp $HOME/.globus/usercert.pem /tmp/$USER/globus/
Line: 32 to 30
 setenv X509_USER_CER /tmp/$USER/globus/usercert.pem setenv X509_USER_KEY /tmp/$USER/globus/userkey.pem ## Make new .glite directory and export it:
Changed:
<
<
if ( -d /tmp/$USER/glite ) then chmod -R 755 /tmp/$USER/glite endif
>
>
chmod -R 755 /tmp/$USER/glite >&/dev/null
 rm -rf /tmp/$USER/glite mkdir -p /tmp/$USER/glite
Changed:
<
<
cp -rd $HOME/.glite/vomses /tmp/$USER/glite/
>
>
cp -rd $HOME/.glite/* /tmp/$USER/glite/
 setenv GLITE_USER_HOME /tmp/$USER/glite (for tcsh).
Changed:
<
<
This ensures that the necessary files are accessible even after your AFS tickets expire.
>
>
This ensures that the necessary files are accessible even after your AFS tickets expire (as they are copied to some local directory).
 

Simple job submission with traditional JDL files

Revision 102008-11-04 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Changed:
<
<
Note: On certain platforms, file or directory names, given as command line arguments to wmsx-provider.sh or wmsx-requestor.sh, are are only recognized properly, when specified as absolute paths.
>
>
Note: If your home directory is located on AFS, it is advised to source the following shell script before logging on to the Grid:
## Make new .globus directory and export it:
if [[ -d /tmp/$USER/globus ]] ; then chmod -R 755 /tmp/$USER/globus ; fi
rm -rf /tmp/$USER/globus
mkdir -p /tmp/$USER/globus
cp $HOME/.globus/usercert.pem /tmp/$USER/globus/
cp $HOME/.globus/userkey.pem /tmp/$USER/globus/
export X509_USER_CER=/tmp/$USER/globus/usercert.pem
export X509_USER_KEY=/tmp/$USER/globus/userkey.pem
## Make new .glite directory and export it:
if [[ -d /tmp/$USER/glite ]] ; then chmod -R 755 /tmp/$USER/glite ; fi
rm -rf /tmp/$USER/glite
mkdir -p /tmp/$USER/glite
cp -rd $HOME/.glite/vomses /tmp/$USER/glite/
export GLITE_USER_HOME=/tmp/$USER/glite
(for bash), or
## Make new .globus directory and export it:
if ( -d /tmp/$USER/globus ) then
    chmod -R 755 /tmp/$USER/globus
endif
rm -rf /tmp/$USER/globus
mkdir -p /tmp/$USER/globus
cp $HOME/.globus/usercert.pem /tmp/$USER/globus/
cp $HOME/.globus/userkey.pem /tmp/$USER/globus/
setenv X509_USER_CER /tmp/$USER/globus/usercert.pem
setenv X509_USER_KEY /tmp/$USER/globus/userkey.pem
## Make new .glite directory and export it:
if ( -d /tmp/$USER/glite ) then
    chmod -R 755 /tmp/$USER/glite
endif
rm -rf /tmp/$USER/glite
mkdir -p /tmp/$USER/glite
cp -rd $HOME/.glite/vomses /tmp/$USER/glite/
setenv GLITE_USER_HOME /tmp/$USER/glite
(for tcsh). This ensures that the necessary files are accessible even after your AFS tickets expire.
 

Simple job submission with traditional JDL files

Changed:
<
<
Job submission with edg-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.
>
>
Job submission with glite-wms-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.
  You can find a prepared example for this task here: simplejob.tar.gz. To submit the example job, type the following commands.
Line: 21 to 60
 
> voms-proxy-init  -voms your_vo
# If your job is expected to be long, also get long term authentication:
Changed:
<
<
> myproxy-init -n
>
>
> myproxy-init -d -n
 

Submit the job:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
Deleted:
<
<
# If you want to use a backend other than glitewms (possibilities: edg/glite/glitewms/worker): > wmsx-requestor.sh -backend your_backend
 # If your job is expected to be long, also make the WMSX remember your grid password: > wmsx-requestor.sh -remembergrid
# If your job is expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
Line: 44 to 81
 If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Added:
>
>
> killall java # <- This is necessary due to a WMSX bug.
 # If myproxy-init was used, i.e. for long term jobs:
Changed:
<
<
> myproxy-destroy
>
>
> myproxy-destroy -d
 > voms-proxy-destroy
Line: 68 to 106
 
> voms-proxy-init  -voms your_vo
# If your jobs are expected to be long, also get long term authentication:
Changed:
<
<
> myproxy-init -n
>
>
> myproxy-init -d -n
 

Submit the jobs:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
Deleted:
<
<
# If you want to use a backend other than glitewms (possibilities: edg/glite/glitewms/worker): > wmsx-requestor.sh -backend your_backend
 # If your jobs are expected to be long, also make the WMSX remember your grid password: > wmsx-requestor.sh -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
Line: 91 to 127
 If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Added:
>
>
> killall java # <- This is necessary due to a WMSX bug.
 # If myproxy-init was used, i.e. for long term jobs:
Changed:
<
<
> myproxy-destroy
>
>
> myproxy-destroy -d
 > voms-proxy-destroy
Line: 113 to 150
 
> voms-proxy-init  -voms your_vo
# If your jobs are expected to be long, also get long term authentication:
Changed:
<
<
> myproxy-init -n
>
>
> myproxy-init -d -n
 

Submit the jobs:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
Deleted:
<
<
# If you want to use a backend other than glitewms (possibilities: edg/glite/glitewms/worker): > wmsx-requestor.sh -backend your_backend
 # If your jobs are expected to be long, also make the WMSX remember your grid password: > wmsx-requestor.sh -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
Line: 136 to 171
 If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Added:
>
>
> killall java # <- This is necessary due to a WMSX bug.
 # If myproxy-init was used, i.e. for long term jobs:
Changed:
<
<
> myproxy-destroy
>
>
> myproxy-destroy -d
 > voms-proxy-destroy
Changed:
<
<
-- AndrasLaszlo - 16 Apr 2008
>
>
-- AndrasLaszlo - 04 Nov 2008
 

META FILEATTACHMENT attr="" autoattached="1" comment="Interactive example" date="1208360461" name="interactive.tar.gz" path="interactive.tar.gz" size="529" user="Main.AndrasLaszlo" version="1"

Revision 92008-06-24 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 28 to 28
 
> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
Added:
>
>
# If you want to use a backend other than glitewms (possibilities: edg/glite/glitewms/worker): > wmsx-requestor.sh -backend your_backend
 # If your job is expected to be long, also make the WMSX remember your grid password: > wmsx-requestor.sh -remembergrid
# If your job is expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
Line: 73 to 75
 
> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
Added:
>
>
# If you want to use a backend other than glitewms (possibilities: edg/glite/glitewms/worker): > wmsx-requestor.sh -backend your_backend
 # If your jobs are expected to be long, also make the WMSX remember your grid password: > wmsx-requestor.sh -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
Line: 116 to 120
 
> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
Added:
>
>
# If you want to use a backend other than glitewms (possibilities: edg/glite/glitewms/worker): > wmsx-requestor.sh -backend your_backend
 # If your jobs are expected to be long, also make the WMSX remember your grid password: > wmsx-requestor.sh -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:

Revision 82008-04-26 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Added:
>
>
Note: On certain platforms, file or directory names, given as command line arguments to wmsx-provider.sh or wmsx-requestor.sh, are are only recognized properly, when specified as absolute paths.
 

Simple job submission with traditional JDL files

Job submission with edg-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.

Revision 72008-04-16 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 6 to 6
  Job submission with edg-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.
Changed:
<
<
You can find a prepared example for this task here: . To submit the example job, type the following commands.
>
>
You can find a prepared example for this task here: simplejob.tar.gz. To submit the example job, type the following commands.
  Get the tarball:
Line: 45 to 45
 > voms-proxy-destroy
Added:
>
>
An educative example for simple interactive job submission may be found here: interactive.tar.gz. This works in a similar way as the simplejob.tar.gz example.
 

Mass submission of independent jobs: parameter scan with short calculations

WMSX is a very convenient tool for mass submission of short, e.g. of about maximum 1 day lifetime, independent jobs. Possible application: parameter scan studies with relatively short calculations. (The job lifetime is limited by the typical allowed maximal job lifetimes, assigned by the Grid sites. A limit of 3 day running time is typical, so a maximum 1 day job is fine.) WMSX automatically manages output retrieval issues and the limiting of the number of concurrently running jobs. For this purpose, the jobs are described with a slightly extended JDL language, which is preprocessed by WMSX. If JobType = "Interactive" is set, the file containing StdOut / StdErr is updated on the fly, so you can see what your job is actually doing. A pre-execution script helps to prepare inputs, and a post-execution script helps to process outputs of the job.

Changed:
<
<
You can find a prepared example for this task here: . To submit the example job, type the following commands.
>
>
You can find a prepared example for this task here: sample.tar.gz. To submit the example job, type the following commands.
  Get the tarball:
Line: 92 to 94
  The mass submission has an important feature: each job can decide whether a launch of a further job is needed. This decision is done by the post-execution script: if it returns with exitcode 1, the chain script is invoked. The lines of the standard output of the chain script is interpreted by WMSX as lines in the arg.list file (see previous example), so new jobs are launched. In this way, jobs can be chained (and the chains can also fork, as a job can also launch multiple jobs). In this way, a parameter study with long term calculations can be performed. The long term calculations are preformed by the job chains. A good example is numerical solution of partial differential equations: multiple chains are launched (possibly with different initial conditions), and each chain is imitating a long term job (the solution of the partial differential equation with a given initial condition), split up into subsequent shorter term jobs.
Changed:
<
<
You can find a prepared example for this task here: . To submit the example job, type the following commands. (Similar to the previous example.)
>
>
You can find a prepared example for this task here: chainsample.tar.gz. To submit the example job, type the following commands. (Similar to the previous example.)
  Get the tarball:
Line: 130 to 132
 > myproxy-destroy
> voms-proxy-destroy
\ No newline at end of file
Added:
>
>

-- AndrasLaszlo - 16 Apr 2008

META FILEATTACHMENT attr="" autoattached="1" comment="Interactive example" date="1208360461" name="interactive.tar.gz" path="interactive.tar.gz" size="529" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Simplejob example" date="1208360425" name="simplejob.tar.gz" path="simplejob.tar.gz" size="501" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Sample example" date="1208360496" name="sample.tar.gz" path="sample.tar.gz" size="1340" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Chainsample example" date="1208360521" name="chainsample.tar.gz" path="chainsample.tar.gz" size="1653" user="Main.AndrasLaszlo" version="1"

Revision 62007-12-18 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 45 to 45
 > voms-proxy-destroy
Changed:
<
<

Mass submission of independent jobs: parameter scan with short (e.g. 1 day) calculations

>
>

Mass submission of independent jobs: parameter scan with short calculations

 
Changed:
<
<
WMSX is a very convenient tool for mass submission of independent jobs. Possible application: parameter scan studies. It automatically manages output retrieval issues and the limiting of the number of concurrently running jobs. For this purpose, the jobs are described with a slightly extended JDL language, which is preprocessed by WMSX. If JobType = "Interactive" is set, the file containing StdOut / StdErr is updated on the fly, so you can see what your job is actually doing. A pre-execution script helps to prepare inputs, and a post-execution script helps to process outputs of the job.
>
>
WMSX is a very convenient tool for mass submission of short, e.g. of about maximum 1 day lifetime, independent jobs. Possible application: parameter scan studies with relatively short calculations. (The job lifetime is limited by the typical allowed maximal job lifetimes, assigned by the Grid sites. A limit of 3 day running time is typical, so a maximum 1 day job is fine.) WMSX automatically manages output retrieval issues and the limiting of the number of concurrently running jobs. For this purpose, the jobs are described with a slightly extended JDL language, which is preprocessed by WMSX. If JobType = "Interactive" is set, the file containing StdOut / StdErr is updated on the fly, so you can see what your job is actually doing. A pre-execution script helps to prepare inputs, and a post-execution script helps to process outputs of the job.
  You can find a prepared example for this task here: . To submit the example job, type the following commands.

Revision 52007-11-30 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 45 to 45
 > voms-proxy-destroy
Changed:
<
<

Mass submission of independent jobs: simple parameter scan

>
>

Mass submission of independent jobs: parameter scan with short (e.g. 1 day) calculations

  WMSX is a very convenient tool for mass submission of independent jobs. Possible application: parameter scan studies. It automatically manages output retrieval issues and the limiting of the number of concurrently running jobs. For this purpose, the jobs are described with a slightly extended JDL language, which is preprocessed by WMSX. If JobType = "Interactive" is set, the file containing StdOut / StdErr is updated on the fly, so you can see what your job is actually doing. A pre-execution script helps to prepare inputs, and a post-execution script helps to process outputs of the job.
Line: 88 to 88
 > voms-proxy-destroy
Changed:
<
<

Mass submission of chained jobs: parameter scan with chained jobs

>
>

Mass submission of chained jobs: parameter scan with long calculations

  The mass submission has an important feature: each job can decide whether a launch of a further job is needed. This decision is done by the post-execution script: if it returns with exitcode 1, the chain script is invoked. The lines of the standard output of the chain script is interpreted by WMSX as lines in the arg.list file (see previous example), so new jobs are launched. In this way, jobs can be chained (and the chains can also fork, as a job can also launch multiple jobs). In this way, a parameter study with long term calculations can be performed. The long term calculations are preformed by the job chains. A good example is numerical solution of partial differential equations: multiple chains are launched (possibly with different initial conditions), and each chain is imitating a long term job (the solution of the partial differential equation with a given initial condition), split up into subsequent shorter term jobs.

Revision 42007-11-29 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Changed:
<
<

Simple job

>
>

Simple job submission with traditional JDL files

  Job submission with edg-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.
Line: 47 to 47
 

Mass submission of independent jobs: simple parameter scan

Added:
>
>
WMSX is a very convenient tool for mass submission of independent jobs. Possible application: parameter scan studies. It automatically manages output retrieval issues and the limiting of the number of concurrently running jobs. For this purpose, the jobs are described with a slightly extended JDL language, which is preprocessed by WMSX. If JobType = "Interactive" is set, the file containing StdOut / StdErr is updated on the fly, so you can see what your job is actually doing. A pre-execution script helps to prepare inputs, and a post-execution script helps to process outputs of the job.

You can find a prepared example for this task here: . To submit the example job, type the following commands.

Get the tarball:

> tar  -xzf  sample.tar.gz
> rm  -f  sample.tar.gz
> cd  sample

Log onto the Grid:

> voms-proxy-init  -voms your_vo
# If your jobs are expected to be long, also get long term authentication:
> myproxy-init  -n

Submit the jobs:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
# If your jobs are expected to be long, also make the WMSX remember your grid password:
> wmsx-requestor.sh  -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
> wmsx-requestor.sh  -rememberafs
# If you want to submit many-many jobs, limit the number of concurrently running jobs e.g. to 100:
> wmsx-requestor.sh  -n 100
> wmsx-requestor.sh  -a arg.list

The file arg.list contains a list of jobs (together with its parameters) to be started. Your results shall be retrieved under the directory workdir/out. If your job was interactive (JobType = "Interactive" was set in the JDL file), the file containting StdOut / StdErr shall be updated on the fly.

If you are finished, you can stop WMSX, and destroy your Grid authentications:

> wmsx-requestor.sh  -k
# If myproxy-init was used, i.e. for long term jobs:
> myproxy-destroy
> voms-proxy-destroy
 

Mass submission of chained jobs: parameter scan with chained jobs

Added:
>
>
The mass submission has an important feature: each job can decide whether a launch of a further job is needed. This decision is done by the post-execution script: if it returns with exitcode 1, the chain script is invoked. The lines of the standard output of the chain script is interpreted by WMSX as lines in the arg.list file (see previous example), so new jobs are launched. In this way, jobs can be chained (and the chains can also fork, as a job can also launch multiple jobs). In this way, a parameter study with long term calculations can be performed. The long term calculations are preformed by the job chains. A good example is numerical solution of partial differential equations: multiple chains are launched (possibly with different initial conditions), and each chain is imitating a long term job (the solution of the partial differential equation with a given initial condition), split up into subsequent shorter term jobs.

You can find a prepared example for this task here: . To submit the example job, type the following commands. (Similar to the previous example.)

Get the tarball:

> tar  -xzf  chainsample.tar.gz
> rm  -f  chainsample.tar.gz
> cd  chainsample

Log onto the Grid:

> voms-proxy-init  -voms your_vo
# If your jobs are expected to be long, also get long term authentication:
> myproxy-init  -n
 
Added:
>
>
Submit the jobs:
> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
# If your jobs are expected to be long, also make the WMSX remember your grid password:
> wmsx-requestor.sh  -remembergrid
# If your jobs are expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
> wmsx-requestor.sh  -rememberafs
# If you want to submit many-many jobs, limit the number of concurrently running jobs e.g. to 100:
> wmsx-requestor.sh  -n 100
> wmsx-requestor.sh  -a arg.list
 
Added:
>
>
The file arg.list contains a list of jobs (together with its parameters) to be started. Your results shall be retrieved under the directory workdir/out. If your job was interactive (JobType = "Interactive" was set in the JDL file), the file containting StdOut / StdErr shall be updated on the fly.
 
Changed:
<
<
-- AndrasLaszlo - 28 Nov 2007
>
>
If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
# If myproxy-init was used, i.e. for long term jobs:
> myproxy-destroy
> voms-proxy-destroy

Revision 32007-11-29 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX examples

Line: 30 to 30
 > wmsx-requestor.sh -remembergrid
# If your job is expected to be long and you are working on AFS, also make the WMSX remember your AFS password: > wmsx-requestor.sh -rememberafs
Changed:
<
<
> wmsx-requestor.sh -j simplejob.jdl -o workdir/out/StdOut -r workdir/out/resultdir
>
>
> wmsx-requestor.sh -j simplejob.jdl -r workdir/out/resultdir
# Or, if JobType = "Interactive" was set in the JDL file, say instead: > wmsx-requestor.sh -j simplejob.jdl -r workdir/out/resultdir -o workdir/out/StdOutFile
 
Changed:
<
<
Your results shall be retrieved into the directory workdir/out/resultdir, and the file containting StdOut / StdErr shall be workdir/out/StdOut, which is retrieved on the fly, if JobType = "Interactive" was set in your JDL file.
>
>
Your results shall be retrieved into the directory workdir/out/resultdir. If your job was interactive (JobType = "Interactive" was set in the JDL file), the file containting StdOut / StdErr shall be workdir/out/StdOutFile, which shall be updated on the fly. Remark: if the JobType = "Interactive" was set, the fields StdInput, StdOutput and StdError cannot be set in the JDL file.
  If you are finished, you can stop WMSX, and destroy your Grid authentications:
> wmsx-requestor.sh  -k
Changed:
<
<
# If myproxy-init was used:
>
>
# If myproxy-init was used, i.e. for long term jobs:
 > myproxy-destroy
> voms-proxy-destroy
Line: 47 to 49
 

Mass submission of chained jobs: parameter scan with chained jobs

Deleted:
<
<

A real-life example

 

Revision 22007-11-28 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"
Changed:
<
<

WMSX Examples

>
>

WMSX examples

 

Simple job

Added:
>
>
Job submission with edg-job-submit is not very straightforward, as the generated job identifier has to be stored, and then the job output retrieval directory has to be resolved by this identifier. To save you from this typing exercise, the WMSX is capable of managing the output retrieval directory for simple jobs.

You can find a prepared example for this task here: . To submit the example job, type the following commands.

Get the tarball:

> tar  -xzf  simplejob.tar.gz
> rm  -f  simplejob.tar.gz
> cd  simplejob

Log onto the Grid:

> voms-proxy-init  -voms your_vo
# If your job is expected to be long, also get long term authentication:
> myproxy-init  -n

Submit the job:

> wmsx-provider.sh  workdir
> wmsx-requestor.sh  -vo your_vo
# If your job is expected to be long, also make the WMSX remember your grid password:
> wmsx-requestor.sh  -remembergrid
# If your job is expected to be long and you are working on AFS, also make the WMSX remember your AFS password:
> wmsx-requestor.sh  -rememberafs
> wmsx-requestor.sh  -j simplejob.jdl  -o workdir/out/StdOut  -r workdir/out/resultdir

Your results shall be retrieved into the directory workdir/out/resultdir, and the file containting StdOut / StdErr shall be workdir/out/StdOut, which is retrieved on the fly, if JobType = "Interactive" was set in your JDL file.

If you are finished, you can stop WMSX, and destroy your Grid authentications:

> wmsx-requestor.sh  -k
# If myproxy-init was used:
> myproxy-destroy
> voms-proxy-destroy
 

Mass submission of independent jobs: simple parameter scan

Mass submission of chained jobs: parameter scan with chained jobs

Revision 12007-11-28 - AndrasLaszlo

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WmsxSoftware"

WMSX Examples

Simple job

Mass submission of independent jobs: simple parameter scan

Mass submission of chained jobs: parameter scan with chained jobs

A real-life example

-- AndrasLaszlo - 28 Nov 2007

 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback