Difference: WmsxReference (1 vs. 15)

Revision 152009-07-10 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 39 to 39
 
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running. It only works if you do not quit the interactive terminal (when you quit interactive session, AFS automatically deletes your tokens).
-forgetafs
Forget the AFS password.
-vo yourvo
Set the VO for all future job submissions to yourvo.
Changed:
<
<
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs. If your grid key files (userkey.pem and usercert.pem, typically located under ~/.globus) are located on AFS space, you should use -rememberafs, and should not quit the interactive terminal. If this is not feasible, copy the .globus directory to some local disk space, and point the environmental variables X509_USER_CER and X509_USER_KEY to the absolute path of usercert.pem and userkey.pem, respectively, and also copy your ~/.glite directory to some local disk space, and point the environmental variable GLITE_USER_HOME to the absolute path of that directory.
>
>
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs. If your grid key files (userkey.pem and usercert.pem, typically located under ~/.globus) are located on AFS space, you should use -rememberafs, and should not quit the interactive terminal. If this is not feasible, copy the .globus directory to some local disk space, and point the environmental variables X509_USER_CER, X509_USER_KEY and X509_CERT_DIR to the absolute path of usercert.pem, userkey.pem files and certificates directory, respectively, and also copy your ~/.glite directory to some local disk space, and point the environmental variable GLITE_USER_HOME to the absolute path of that directory.
  The possible job submission procedures are described in the next sections.
Line: 117 to 117
 If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.

The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.) \ No newline at end of file

Added:
>
>

The backend concept

The backend is the system which is actually taking care about your job submission. This section needs expansion...

Graphical User Interface

A simple graphical interface is written to help job flow monitoring. Start with:

wmsx-gui.sh

-- AndrasLaszlo - 10 Jul 2009

Revision 142009-07-10 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 35 to 35
 
-f
Check if the provider is running.
-k
Kill provider.
Changed:
<
<
-backend yourbackend
With this, you may specify the job submission backend. Here, yourbackend may be any of edg, glite, glitewms, worker. Default backend is glitewms.
>
>
-backend yourbackend
With this, you may specify the job submission backend. Here, yourbackend may be any of glitewms, edg, worker, or local, fake, gat. Default backend is glitewms. The backend is the system which is actually taking care about your job submission. The backend concept shall be explained later.
 
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running. It only works if you do not quit the interactive terminal (when you quit interactive session, AFS automatically deletes your tokens).
-forgetafs
Forget the AFS password.
-vo yourvo
Set the VO for all future job submissions to yourvo.
Line: 94 to 94
  If the WMSX JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).
Changed:
<
<
In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension).
>
>
In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.wjdl file (or if not present, to the the COMMAND.tar.gz file), without the ".wjdl" extension (or without the ".tar.gz" extension).
 

Pre-execution and post-execution scripts

Line: 106 to 106
  The retrieved outputs of your job will always be in the tarball out.tar.gz under the generated job output directory.
Added:
>
>
If COMMAND_preexec returns with 0 nothing further happens: the Grid job is submitted. If returns with 1, the actual Grid job shall not be launched. This feature can be used for a job submission decision.
 If COMMAND_postexec returns with 0 nothing further happens. If returns with 1, the script COMMAND_chain is called, if present, which can be used to launch further jobs.

Job chaining

Revision 132008-11-04 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 39 to 39
 
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running. It only works if you do not quit the interactive terminal (when you quit interactive session, AFS automatically deletes your tokens).
-forgetafs
Forget the AFS password.
-vo yourvo
Set the VO for all future job submissions to yourvo.
Changed:
<
<
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs. If your grid key files (userkey.pem and usercert.pem, typically located under ~/.globus) are located on AFS space, you should use -rememberafs, and should not quit the interactive terminal. If this is not feasible, copy the .globus directory to some local disk space, and point the environmental variables X509_USER_CER and X509_USER_KEY to the absolute path of usercert.pem and userkey.pem, respectively.
>
>
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs. If your grid key files (userkey.pem and usercert.pem, typically located under ~/.globus) are located on AFS space, you should use -rememberafs, and should not quit the interactive terminal. If this is not feasible, copy the .globus directory to some local disk space, and point the environmental variables X509_USER_CER and X509_USER_KEY to the absolute path of usercert.pem and userkey.pem, respectively, and also copy your ~/.glite directory to some local disk space, and point the environmental variable GLITE_USER_HOME to the absolute path of that directory.
  The possible job submission procedures are described in the next sections.

Revision 122008-06-24 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 35 to 35
 
-f
Check if the provider is running.
-k
Kill provider.
Added:
>
>
-backend yourbackend
With this, you may specify the job submission backend. Here, yourbackend may be any of edg, glite, glitewms, worker. Default backend is glitewms.
 
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running. It only works if you do not quit the interactive terminal (when you quit interactive session, AFS automatically deletes your tokens).
-forgetafs
Forget the AFS password.
-vo yourvo
Set the VO for all future job submissions to yourvo.
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs. If your grid key files (userkey.pem and usercert.pem, typically located under ~/.globus) are located on AFS space, you should use -rememberafs, and should not quit the interactive terminal. If this is not feasible, copy the .globus directory to some local disk space, and point the environmental variables X509_USER_CER and X509_USER_KEY to the absolute path of usercert.pem and userkey.pem, respectively.
Changed:
<
<
The possible job submission procedures ars described in the next sections.
>
>
The possible job submission procedures are described in the next sections.
 

Submission of single jobs with traditional JDL files

Line: 74 to 75
 
-name runname
Give the name runname to this execution run (optional). It is used as a subdirectory under the working directory of the provider to distinguish between different ArgList submissions.
-n nmaxjobs
Set the maximal number of concurrently running jobs to nmaxjobs (optional).
Changed:
<
<
The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl, which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed, which shall be discussed in the followings.
>
>
The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.wjdl, which describes your job. If the COMMAND.wjdl is not present, default job specifications are assumed, which shall be discussed in the followings.
  The outputs of the jobs are written into generated directories under the out directory of the specified working directory of the provider.

WMSX JDL files

Changed:
<
<
For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job.
>
>
For each COMMAND in the ArgList file, there may be a WMSX JDL file COMMAND.wjdl, to customize the properties of your job.
 The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:

JobType
If this variable is set to "Interactive", the jobs will be run as interactively in the sense that the StdOut / StdError is retrieved on-the-fly, so you are able to see what your job is currently doing. If not set, the job is not interactive by default.
Line: 91 to 92
 
Software
List of software that must be present (executable) on the target machine. The special key "AFS" requires and checks AFS presence. E.g.: Software = {"AFS", "g++"};. If not set, defaults to empty.
Requirements
Extra queue requirements, like in a traditional JDL file. If not set, defaults to empty.
Changed:
<
<
If the JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).
>
>
If the WMSX JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).
  In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension).

Revision 112008-04-26 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 21 to 21
  After the start of the provider, directories log, debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. The files jobids.all, jobids.running, jobids.failed and jobids.done under the working directory contain the unique job identifiers of the sent Grid jobs.
Added:
>
>
Note: On certain platforms, the file and directory names, given as command line arguments to wmsx-provider.sh or wmsx-requestor.sh, are only properly recognized when specified as absolute paths.
 

Requestor

Requestor is an application to submit commands to the Provider system for execution. Start with:

Revision 102008-04-07 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 97 to 97
  In most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be called COMMAND_preexec and COMMAND_postexec. They must be executable. They will be run directly before submission and after job output is retrieved, respectively.
Changed:
<
<
COMMAND_preexec, if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments.
>
>
COMMAND_preexec, if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framework), and with all the given arguments from ArgList as following arguments.
  COMMAND_postexec, if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framework), and with all the given arguments from ArgList as following arguments.
Changed:
<
<
The retrieved outputs of your job will always be in the tarball out.tar.gz under the job output directory.
>
>
The retrieved outputs of your job will always be in the tarball out.tar.gz under the generated job output directory.
  If COMMAND_postexec returns with 0 nothing further happens. If returns with 1, the script COMMAND_chain is called, if present, which can be used to launch further jobs.

Revision 92007-12-04 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 33 to 33
 
-f
Check if the provider is running.
-k
Kill provider.
Changed:
<
<
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running.
-forgetafs
Forget the password.
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs.
>
>
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running. It only works if you do not quit the interactive terminal (when you quit interactive session, AFS automatically deletes your tokens).
-forgetafs
Forget the AFS password.
 
-vo yourvo
Set the VO for all future job submissions to yourvo.
Added:
>
>
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs. If your grid key files (userkey.pem and usercert.pem, typically located under ~/.globus) are located on AFS space, you should use -rememberafs, and should not quit the interactive terminal. If this is not feasible, copy the .globus directory to some local disk space, and point the environmental variables X509_USER_CER and X509_USER_KEY to the absolute path of usercert.pem and userkey.pem, respectively.
  The possible job submission procedures ars described in the next sections.

Revision 82007-11-30 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 12 to 12
 Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with:
Changed:
<
<
wmsx-provider.sh -v myworkdir
>
>
wmsx-provider.sh myworkdir [-v]
 

where:

Deleted:
<
<
-v
Specifies that you want debug output on the console. Leave this out once you feel comfortable.
 
myworkdir
Location of the working directory. This is where all the logging and output goes. You may want to set this to a location inside your home directory.
Added:
>
>
-v
Specifies that you want debug output on the console. Leave this out once you feel comfortable.
  After the start of the provider, directories log, debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. The files jobids.all, jobids.running, jobids.failed and jobids.done under the working directory contain the unique job identifiers of the sent Grid jobs.
Line: 29 to 29
 wmsx-requestor.sh -option
Changed:
<
<
A complete list of options is available with -h. Some options are:
>
>
A complete list of options is available with -h. Some options are:
 
-f
Check if the provider is running.
-k
Kill provider.
Line: 81 to 81
 For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job. The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:
Added:
>
>
JobType
If this variable is set to "Interactive", the jobs will be run as interactively in the sense that the StdOut / StdError is retrieved on-the-fly, so you are able to see what your job is currently doing. If not set, the job is not interactive by default.
 
Archive
Name of the program archive file. This is the name of the tarball, containing your program. If not specified, defaults to "COMMAND.tar.gz". (Must be of tar-gz format!)
ProgramDir
Name of the root directory inside the program archive file. The files and directories of your program archive are assumed to be under this directory, and their paths are assumed to be given relative to this directory. Setting this to "." means that the content of your tarball is not wrapped in a directory. If not specified, defaults to "COMMAND".
Executable
Name of the executable to run inside the ProgramDir. If not specified, defaults to "COMMAND".
OutputDirectory
The name of the directory under ProgramDir, where the output of your program is written. This is the directory, which is going to be retrieved by the framework as output. If not specified, defaults to "out".
Deleted:
<
<
JobType
If this variable is set to "Interactive", the jobs will be run as interactively in the sense that the StdOut / StdError is retrieved on-the-fly, so you are able to see what your job is currently doing. If not set, the job is not interactive by default.
 
Software
List of software that must be present (executable) on the target machine. The special key "AFS" requires and checks AFS presence. E.g.: Software = {"AFS", "g++"};. If not set, defaults to empty.
Requirements
Extra queue requirements, like in a traditional JDL file. If not set, defaults to empty.

Revision 72007-11-29 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 45 to 45
 Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is:
Changed:
<
<
wmsx-requestor.sh -j example.jdl -o StdOutFile -r resultDir
>
>
wmsx-requestor.sh -j example.jdl -r resultDir [ -o StdOutFile ]
 

Where the options are:

-j
Name of the JDL file.
Deleted:
<
<
-o
File to store StdOut / StdError. If the JDL file has JobType set to "Interactive", then StdOut / StdError will be retrieved while the Job is running and stored in the given filename.
 
-r
When the job is done, results are retrieved and stored in the given directory.
Added:
>
>
-o
If the JDL file has JobType set to "Interactive", then StdOut / StdError will be retrieved while the Job is running and stored in the given filename.
 

Automated mass submission of jobs via ArgList files

Revision 62007-09-18 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Basic usage

Added:
>
>
This is a program for mass job management on the Grid.
 There are two program you will be using: Provider and Requestor.

Provider

Line: 18 to 19
 
-v
Specifies that you want debug output on the console. Leave this out once you feel comfortable.
myworkdir
Location of the working directory. This is where all the logging and output goes. You may want to set this to a location inside your home directory.
Changed:
<
<
After the start of the provider, directories log, debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory.
>
>
After the start of the provider, directories log, debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory. The files jobids.all, jobids.running, jobids.failed and jobids.done under the working directory contain the unique job identifiers of the sent Grid jobs.
 

Requestor

Revision 52007-09-17 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 75 to 75
  The outputs of the jobs are written into generated directories under the out directory of the specified working directory of the provider.
Changed:
<
<

WMSX JDL files.

>
>

WMSX JDL files

  For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job. The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:

Revision 42007-09-16 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 18 to 18
 
-v
Specifies that you want debug output on the console. Leave this out once you feel comfortable.
myworkdir
Location of the working directory. This is where all the logging and output goes. You may want to set this to a location inside your home directory.
Added:
>
>
After the start of the provider, directories log, debug and out is prepared under the working directory. The logging informations of the WMSX actions are written into a file under the log directory.
 

Requestor

Requestor is an application to submit commands to the Provider system for execution. Start with:

Line: 37 to 39
  The possible job submission procedures ars described in the next sections.
Changed:
<
<

Traditional submission of single jobs with traditional JDL files

>
>

Submission of single jobs with traditional JDL files

  Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is:
Line: 71 to 73
  The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl, which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed, which shall be discussed in the followings.
Added:
>
>
The outputs of the jobs are written into generated directories under the out directory of the specified working directory of the provider.
 

WMSX JDL files.

For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job.

Revision 32007-09-15 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 8 to 8
 

Provider

Changed:
<
<
Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with:
>
>
Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with:
 
Changed:
<
<
wmsx-provider.sh -v /tmp/myworkdir
>
>
wmsx-provider.sh -v myworkdir
 

where:

Changed:
<
<
-v
Specifies that you want debug output on the console.
Leave this out once you feel comfortable.
myworkdir
Location of the working directory.
This is where all the output goes. You may want to set this to a location inside your home directory.
>
>
-v
Specifies that you want debug output on the console. Leave this out once you feel comfortable.
myworkdir
Location of the working directory. This is where all the logging and output goes. You may want to set this to a location inside your home directory.
 

Requestor

Changed:
<
<
Requestor is an application to submit commands to the Provider system for execution. Start with:
>
>
Requestor is an application to submit commands to the Provider system for execution. Start with:
 
wmsx-requestor.sh -option
Line: 35 to 30
 
-f
Check if the provider is running.
-k
Kill provider.
Changed:
<
<
-rememberafs
Will ask for your AFS password and renew AFS tokens
until you use.
>
>
-rememberafs
Will ask for your AFS password and renew AFS tokens until provider is running.
 
-forgetafs
Forget the password.
Changed:
<
<
-remembergrid
Will ask for your Grid password and renew the
Grid tokens until there are no more managed jobs.
>
>
-remembergrid
Will ask for your Grid password and renew the Grid tokens until there are no more managed jobs.
 
-vo yourvo
Set the VO for all future job submissions to yourvo.

The possible job submission procedures ars described in the next sections.

Changed:
<
<

Traditional Submission of Single Jobs with Traditional JDL Files

>
>

Traditional submission of single jobs with traditional JDL files

 
Changed:
<
<
Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is:
>
>
Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is:
 
Changed:
<
<
wmsx-requestor.sh -j example.jdl -o /tmp/StdOut -r /tmp/resultDir
>
>
wmsx-requestor.sh -j example.jdl -o StdOutFile -r resultDir
 

Where the options are:

-j
Name of the JDL file.
Changed:
<
<
-o
If the JDL file has JobType set to "Interactive",
then StdOut / StdError will be retrieved while the Job is running and stored in the given filename.
-r
When the job is done, results are retrieved and stored in the
given directory.
>
>
-o
File to store StdOut / StdError. If the JDL file has JobType set to "Interactive", then StdOut / StdError will be retrieved while the Job is running and stored in the given filename.
-r
When the job is done, results are retrieved and stored in the given directory.
 
Changed:
<
<

Automated Mass Submission of Jobs via ArgList Files

>
>

Automated mass submission of jobs via ArgList files

 
Changed:
<
<
By using ArgList files, one can submit many independent jobs and handle their outputs in a simple and efficient way.
>
>
By using ArgList files, one can submit many independent jobs and handle their outputs in a simple and efficient way.
 
Changed:
<
<
The ArgList file can contain lines of the following format:
>
>
The ArgList file can contain lines of the following format:
 
COMMAND parameters
Changed:
<
<
where COMMAND refers to a job name, and parameters are the command line arguments of the executable of your job.
>
>
where COMMAND refers to a job name, and parameters are the command line arguments of the executable of your job.
  To submit the jobs, use:
Changed:
<
<
-a args.file
Submit many jobs with the ArgList file args.file.
-name runname
Give the name runname to this execution run (optional).
It is used as a subdirectory under the working directory to distinguish between different ArgList submissions.
-n nmaxjobs
Set the maximal number of concurrently running jobs
to nmaxjobs (optional).

The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl, which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed.

>
>
-a args.file
Submit many jobs with the ArgList file, named args.file.
-name runname
Give the name runname to this execution run (optional). It is used as a subdirectory under the working directory of the provider to distinguish between different ArgList submissions.
-n nmaxjobs
Set the maximal number of concurrently running jobs to nmaxjobs (optional).

The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl, which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed, which shall be discussed in the followings.

 

WMSX JDL files.

Deleted:
<
<
For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job. The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:

Archive
Name of the program archive file. This is the name of the
tarball, containing your program. If not specified, defaults to "COMMAND.tar.gz". (Must be of tar-gz format!)
ProgramDir
Name of the root directory inside the program archive
file. The files and directories of your program archive are assumed to be under this directory. Setting it to "." means that they are not wrapped in a directory. If not specified, defaults to "COMMAND".
Executable
Name of the executable to run inside the ProgramDir.
If not specified, Defaults to "COMMAND".
OutputDirectory
The name of the directory under ProgramDir, where
the output of your program is written. This is the directory, Which is going to be retrieved by the framework as output. If not specified, defaults to "out".
JobType
If this variable is set to "Interactive", the jobs will
be run as interactively in the sense that the StdOut / StdError is retrieved on-the-fly, so you are able to see what your job is currently doing. If not set, the job is not interactive by default.
Software
List of software that must be present (executable) on the
target machine. The special key "AFS" checks for AFS presence. E.g.: Software = {"AFS", "g++"};. If not set, defaults to empty.
Requirements
Extra queue requirements, like in a traditional JDL file.
If not set, defaults to empty.

If the JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).

In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension, if JDL file is not present).

PreExec and PostExec Scripts

In most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be called COMMAND_preexec and COMMAND_postexec. They must be executable. They will be run directly before submission and after job output is retrieved, respectively.

PreExec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments.

PostExec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framwork), and with all the given arguments from ArgList as following arguments.

The retrieved outputs of your job will always be in the tarball OutputDirectory.tar.gz under the job output directory, where OutputDirectory is what you specified in the JDL file (out by default).

If PostExec returns with 0 nothing further happens. If PostExec returns with 1, the user script COMMAND_chain is called, which can be used to launch further jobs.

Job Chaining

The running time of Grid jobs is limited. This is unavoidable for efficient control on resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state should be dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The script COMMAND_chain is the tool to lauch further jobs, when needed.

If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.

The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.)

 \ No newline at end of file
Added:
>
>
For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job. The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:

Archive
Name of the program archive file. This is the name of the tarball, containing your program. If not specified, defaults to "COMMAND.tar.gz". (Must be of tar-gz format!)
ProgramDir
Name of the root directory inside the program archive file. The files and directories of your program archive are assumed to be under this directory, and their paths are assumed to be given relative to this directory. Setting this to "." means that the content of your tarball is not wrapped in a directory. If not specified, defaults to "COMMAND".
Executable
Name of the executable to run inside the ProgramDir. If not specified, defaults to "COMMAND".
OutputDirectory
The name of the directory under ProgramDir, where the output of your program is written. This is the directory, which is going to be retrieved by the framework as output. If not specified, defaults to "out".
JobType
If this variable is set to "Interactive", the jobs will be run as interactively in the sense that the StdOut / StdError is retrieved on-the-fly, so you are able to see what your job is currently doing. If not set, the job is not interactive by default.
Software
List of software that must be present (executable) on the target machine. The special key "AFS" requires and checks AFS presence. E.g.: Software = {"AFS", "g++"};. If not set, defaults to empty.
Requirements
Extra queue requirements, like in a traditional JDL file. If not set, defaults to empty.

If the JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).

In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension).

Pre-execution and post-execution scripts

In most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be called COMMAND_preexec and COMMAND_postexec. They must be executable. They will be run directly before submission and after job output is retrieved, respectively.

COMMAND_preexec, if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments.

COMMAND_postexec, if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framework), and with all the given arguments from ArgList as following arguments.

The retrieved outputs of your job will always be in the tarball out.tar.gz under the job output directory.

If COMMAND_postexec returns with 0 nothing further happens. If returns with 1, the script COMMAND_chain is called, if present, which can be used to launch further jobs.

Job chaining

The running time of Grid jobs is limited. This is unavoidable for efficient controling of resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter subsequent pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state is dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The script COMMAND_chain is the tool to lauch further jobs, when needed.

If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.

The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.)

Revision 22007-09-14 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WmsxSoftware"

WMSX

Line: 8 to 8
 

Provider

Changed:
<
<
is the background process which does the actual work. There can be only one provider runnung on a machine per user. Start with
>
>
Provider is the background process which does the actual work. There can be only one provider running on a machine per user. Start with:
 
Changed:
<
<
./wmsx-provider -v /tmp/myworkdir
>
>
wmsx-provider.sh -v /tmp/myworkdir
 

where:

Changed:
<
<
-v
Specified you want debug output on the console. Leave this out once you feel comfortable.
/tmp/myworkdir
Location of the work directory. This is where all your output goes. You may want to set this to a location inside your home directory.
>
>
-v
Specifies that you want debug output on the console.
Leave this out once you feel comfortable.
myworkdir
Location of the working directory.
This is where all the output goes. You may want to set this to a location inside your home directory.
 

Requestor

Changed:
<
<
is an application to submit commands to the system for execution. Start with:
>
>
Requestor is an application to submit commands to the Provider system for execution. Start with:
 
Changed:
<
<
./wmsx-requestor -option
>
>
wmsx-requestor.sh -option
 

A complete list of options is available with -h. Some options are:

Changed:
<
<
-f
Check if the provider is running
-k
kill provider
-rememberafs
Will ask for your AFS password and renew AFS tokens until you use
-forgetafs
to forget the password
-remembergrid
Will ask for yout Grid password and renew the Grid tokens until there are no more managed jobs
-n
set the maximum number of concurrently running jobs
-vo
set the VO for all future job submissions
>
>
-f
Check if the provider is running.
-k
Kill provider.
-rememberafs
Will ask for your AFS password and renew AFS tokens
until you use.
-forgetafs
Forget the password.
-remembergrid
Will ask for your Grid password and renew the
Grid tokens until there are no more managed jobs.
-vo yourvo
Set the VO for all future job submissions to yourvo.
 
Changed:
<
<

JDL Files

>
>
The possible job submission procedures ars described in the next sections.
 
Changed:
<
<
Can be sumbitted and managed through the framework. A sample usage is:
>
>

Traditional Submission of Single Jobs with Traditional JDL Files

Single jobs may be submitted and managed via the framework, by using traditional JDL files. A sample usage is:

 
Changed:
<
<
./wmsx-requestor -j example.jdl -o /tmp/StdOut -r /tmp/resultDir
>
>
wmsx-requestor.sh -j example.jdl -o /tmp/StdOut -r /tmp/resultDir
 

Where the options are:

Changed:
<
<
-j
name of the JDL file
-o
If the JDL file has "JobType" set to "Interactive", then StdOut / StdError will be retrieved while the Job is running and stored in the given filename
-r
When the job is done, results are retrieved and stored in the given directory.
>
>
-j
Name of the JDL file.
-o
If the JDL file has JobType set to "Interactive",
then StdOut / StdError will be retrieved while the Job is running and stored in the given filename.
-r
When the job is done, results are retrieved and stored in the
given directory.
 
Changed:
<
<

ArgList Files

>
>

Automated Mass Submission of Jobs via ArgList Files

 
Changed:
<
<
Can specify multiple jobs to create one "larger" jpb.
>
>
By using ArgList files, one can submit many independent jobs and handle their outputs in a simple and efficient way.
 
Changed:
<
<
ArgList Format:
>
>
The ArgList file can contain lines of the following format:
 
Added:
>
>
 COMMAND parameters
Added:
>
>
 
Changed:
<
<
./wmsx-requestor options:

-a
args.file

-name
name of this execution. Is used as directory to distinguish different runs with the same commands.

Custom JDL files.

for Each command, there may be a file command.jdl, with custom JDL contents. sup ported are:

Archive
name of the archive, defaults to COMMAND.tar.gz (must be tar-gz format!)
ProgramDir
name of the program dir inside the tar file, defaults to COMMAND
Executable
name of the executable to run inside the program dir, defaults to COMMAND
OutputDirectory
Which directory to retrieve, defaults to "out"
JobType
If set to "Interactive", the jobs will be run as interactive
Software
List of software that must be present (executable) on the target machine. The special key "AFS" checks for AFS presence
Requirements
Extra requirements, as specified by the original JDL spec.

Pre/PostExec

must be calles COMMAND_preexec and COMMAND_postexec. MUST be executable. Will be run directly before submission and after job output is retrieved.

Preexec is called with the name of the command and its full directory as first argument, and with all the given args as following arguments.

Postexec is called with the name of the command and its full directory as first argument, , the name of the output directory as second argument, and with all the given args as following arguments.

The output of your job will always be in a file "out.tar.gz" in the output directory.

If postexec return with 0 nothing further happens. If postexec returns with 1, COMMAND_chain is called.

>
>
where COMMAND refers to a job name, and parameters are the command line arguments of the executable of your job.
 
Changed:
<
<

Chaining

>
>
To submit the jobs, use:
 
Changed:
<
<
The _chain command's output is interpreted just as a line from the args file. It can therefore be used to start further jobs.
>
>
-a args.file
Submit many jobs with the ArgList file args.file.
-name runname
Give the name runname to this execution run (optional).
It is used as a subdirectory under the working directory to distinguish between different ArgList submissions.
-n nmaxjobs
Set the maximal number of concurrently running jobs
to nmaxjobs (optional).

The word COMMAND refers to a WMSX JDL file (not a traditional JDL file), named COMMAND.jdl, which describes your job. If the COMMAND.jdl is not present, default job specifications are assumed.

WMSX JDL files.

For each COMMAND in the ArgList file, there may be a JDL file COMMAND.jdl, to customize the properties of your job. The structure of a WMSX JDL file is similar to traditional JDL files, however the supported variables are only:

Archive
Name of the program archive file. This is the name of the
tarball, containing your program. If not specified, defaults to "COMMAND.tar.gz". (Must be of tar-gz format!)
ProgramDir
Name of the root directory inside the program archive
file. The files and directories of your program archive are assumed to be under this directory. Setting it to "." means that they are not wrapped in a directory. If not specified, defaults to "COMMAND".
Executable
Name of the executable to run inside the ProgramDir.
If not specified, Defaults to "COMMAND".
OutputDirectory
The name of the directory under ProgramDir, where
the output of your program is written. This is the directory, Which is going to be retrieved by the framework as output. If not specified, defaults to "out".
JobType
If this variable is set to "Interactive", the jobs will
be run as interactively in the sense that the StdOut / StdError is retrieved on-the-fly, so you are able to see what your job is currently doing. If not set, the job is not interactive by default.
Software
List of software that must be present (executable) on the
target machine. The special key "AFS" checks for AFS presence. E.g.: Software = {"AFS", "g++"};. If not set, defaults to empty.
Requirements
Extra queue requirements, like in a traditional JDL file.
If not set, defaults to empty.

If the JDL file is not present, the above default values are assumed (i.e. the tarball has to have the name COMMAND.tar.gz etc.).

In the followings, things are more easily explained if the notion of AbsCOMMAND is introduced: this is simply the full path to the file COMMAND.jdl file (or if not present, to the the COMMAND.tar.gz file), without the ".jdl" extension (or without the ".tar.gz" extension, if JDL file is not present).

PreExec and PostExec Scripts

In most times it is useful to have pre-execution and post-execution scripts. These may be used for e.g. preparing the input data files, or archiving output data files etc. If present, these have to be called COMMAND_preexec and COMMAND_postexec. They must be executable. They will be run directly before submission and after job output is retrieved, respectively.

PreExec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, and with all the given arguments from ArgList as following arguments.

PostExec , if present, is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument (automatically generated by the framwork), and with all the given arguments from ArgList as following arguments.

The retrieved outputs of your job will always be in the tarball OutputDirectory.tar.gz under the job output directory, where OutputDirectory is what you specified in the JDL file (out by default).

If PostExec returns with 0 nothing further happens. If PostExec returns with 1, the user script COMMAND_chain is called, which can be used to launch further jobs.

Job Chaining

The running time of Grid jobs is limited. This is unavoidable for efficient control on resources. The time limit depends on the given queue, but a typical value is three days. However, one often faces such computing problems, when the total running time of the jobs cannot be estimated a priori, or it is estimated to be very long. For such cases, the job chaining is the solution: the program has to be split up into shorter pieces with limited running time. The program has to be written in such a way, that its running time is limited internally (e.g. to one day), and when this time limit is exceeded, its last state should be dumped as output. The next copy of the program has to be started with this last state as input, thus, by such chain of limited lifetime jobs, one can imitate arbitrary long lifetime jobs. The script COMMAND_chain is the tool to lauch further jobs, when needed.

If the COMMAND_postexec script returns 1, the script COMMAND_chain is invoked (must be executable). In this case, if present, it is automatically called by the framework with the AbsCOMMAND as first argument, the name of the job output directory as second argument, and with all the given arguments from ArgList as following arguments.

The output of COMMAND_chain is interpreted by the framework as ArgList lines, just as if they were lines from the initial ArgList file. Therefore, it can be used to lauch further jobs by a finished job, depending on the decision of the COMMAND_postexec script. This is called the job chaining. (The COMMAND_chain may have multiple lines as output. Each line is interpreted like a line from the ArgList file, so multiple jobs may also be lauched: the job chain may also fork.)

 \ No newline at end of file

Revision 12007-09-05 - MaxBerger

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WmsxSoftware"

WMSX

Basic usage

There are two program you will be using: Provider and Requestor.

Provider

is the background process which does the actual work. There can be only one provider runnung on a machine per user. Start with

./wmsx-provider -v /tmp/myworkdir

where:

-v
Specified you want debug output on the console. Leave this out once you feel comfortable.
/tmp/myworkdir
Location of the work directory. This is where all your output goes. You may want to set this to a location inside your home directory.

Requestor

is an application to submit commands to the system for execution. Start with:

./wmsx-requestor -option

A complete list of options is available with -h. Some options are:

-f
Check if the provider is running
-k
kill provider
-rememberafs
Will ask for your AFS password and renew AFS tokens until you use
-forgetafs
to forget the password
-remembergrid
Will ask for yout Grid password and renew the Grid tokens until there are no more managed jobs
-n
set the maximum number of concurrently running jobs
-vo
set the VO for all future job submissions

JDL Files

Can be sumbitted and managed through the framework. A sample usage is:

./wmsx-requestor -j example.jdl -o /tmp/StdOut -r /tmp/resultDir

Where the options are:

-j
name of the JDL file
-o
If the JDL file has "JobType" set to "Interactive", then StdOut / StdError will be retrieved while the Job is running and stored in the given filename
-r
When the job is done, results are retrieved and stored in the given directory.

ArgList Files

Can specify multiple jobs to create one "larger" jpb.

ArgList Format:

COMMAND parameters

./wmsx-requestor options:

-a
args.file

-name
name of this execution. Is used as directory to distinguish different runs with the same commands.

Custom JDL files.

for Each command, there may be a file command.jdl, with custom JDL contents. sup ported are:

Archive
name of the archive, defaults to COMMAND.tar.gz (must be tar-gz format!)
ProgramDir
name of the program dir inside the tar file, defaults to COMMAND
Executable
name of the executable to run inside the program dir, defaults to COMMAND
OutputDirectory
Which directory to retrieve, defaults to "out"
JobType
If set to "Interactive", the jobs will be run as interactive
Software
List of software that must be present (executable) on the target machine. The special key "AFS" checks for AFS presence
Requirements
Extra requirements, as specified by the original JDL spec.

Pre/PostExec

must be calles COMMAND_preexec and COMMAND_postexec. MUST be executable. Will be run directly before submission and after job output is retrieved.

Preexec is called with the name of the command and its full directory as first argument, and with all the given args as following arguments.

Postexec is called with the name of the command and its full directory as first argument, , the name of the output directory as second argument, and with all the given args as following arguments.

The output of your job will always be in a file "out.tar.gz" in the output directory.

If postexec return with 0 nothing further happens. If postexec returns with 1, COMMAND_chain is called.

Chaining

The _chain command's output is interpreted just as a line from the args file. It can therefore be used to start further jobs.

 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback