Difference: BasicUsage (1 vs. 15)

Revision 152013-07-24 - AndrasHazi

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Changed:
<
<
On this page, we intend to provide a brief introduction on simple job submission onto the Grid. One can also find a brief introductory material on the Grid homepage of RMKI.
>
>
On this page, we intend to provide a brief introduction on simple job submission onto the Grid. One can also find a brief introductory material on the Grid homepage of RMI.
 

Log onto a User Interface ( _UI) machine

Revision 142010-10-13 - BenceSomhegyi

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 8 to 8
  After you have logged onto a UI machine, you are able to submit commands to the Grid.
Changed:
<
<
You should have your Grid user key files usercert.pem and userkey.pem under ~/.globus directory. If they are located, for whatever reason, under an alternative location, it can be told to the Grid applications by setting the following environmental variables, like:
>
>
You should have your Grid user key files usercert.pem and userkey.pem under ~/.globus directory. If you have a certificate in PKCS#12 (.p12) format, you can convert it to PEM encoded certificate and private key using the following to commands:
openssl pkcs12 -in cert.p12 -clcerts -nokeys -out usercert.pem
openssl pkcs12 -in cert.p12 -nocerts -out userkey.pem

If they are located, for whatever reason, under an alternative location, it can be told to the Grid applications by setting the following environmental variables, like:

  export X509_USER_CER=/some_directory/globus/usercert.pem
Line: 42 to 46
  If you are member of more then one VO, you can choose between them by using the glite-voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:
Changed:
<
<
> glite-voms-proxy-init -voms hungrid Or:
>
>
> voms-proxy-init -voms hungrid Or:

> voms-proxy-init -voms hungrid -valid 04:00

 
Changed:
<
<
> glite-voms-proxy-init -voms hungrid -valid 04:00
>
>
By default, voms-proxy-init generates a proxy valid for 12 hours. You can request a maximum validity of 24 hours. Be aware that if your proxy expires when your jobs are still running (or waiting), they will abort automatically. You should use MyProxy (see below) for long-term jobs, or when submitting a lot of jobs (more than 200).
 
Changed:
<
<
To get information on your user proxy, you can use the commands grid-proxy-info or glite-voms-proxy-info -all. You can destroy your user proxy by grid-proxy-destroy or glite-voms-proxy-destroy.
>
>
To get information on your user proxy, you can use the command voms-proxy-info -all. You can destroy your user proxy by voms-proxy-destroy.
 

Get your jobs authenticated on the Grid

Line: 141 to 147
  For further information, look at the man pages of the above commands, and maybe also to the man pages of other glite-wms- commands. For further references on simple job submission, see https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf. Also a complete description of the JDL language is available there.
Deleted:
<
<
-- AndrasLaszlo - 10 Jul 2009
 \ No newline at end of file
Added:
>
>
-- AndrasLaszlo

-- BenceSomhegyi

Revision 132010-09-13 - BenceSomhegyi

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 101 to 101
  Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:
Changed:
<
<
> glite-wms-job-list-match --vo your_vo testjob.jdl
>
>
> glite-wms-job-list-match -a testjob.jdl
 

This will return a list of Grid queues (computing elements), which are capable of executing your job.

The job can be submitted by the command:

Changed:
<
<
> glite-wms-job-submit --vo your_vo -a testjob.jdl
>
>
> glite-wms-job-submit -a testjob.jdl
 

This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.

Revision 122010-09-10 - BenceSomhegyi

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 40 to 40
 

Log onto the Grid (get authenticated on the Grid)

Deleted:
<
<
This means getting a so called user proxy. Commands are:

> grid-proxy-init Here, you will be prompted for your grid password. Or:

> grid-proxy-init -valid 04:00 This is the same, but the authentication will expire in 4 hours. The default (and maximum) lifetime is 12 hours.

 If you are member of more then one VO, you can choose between them by using the glite-voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:
Changed:
<
<
> glite-voms-proxy-init -voms hungrid Or:
>
>
> glite-voms-proxy-init -voms hungrid Or:
 
Changed:
<
<
> glite-voms-proxy-init -voms hungrid -valid 04:00
>
>
> glite-voms-proxy-init -voms hungrid -valid 04:00
  To get information on your user proxy, you can use the commands grid-proxy-info or glite-voms-proxy-info -all. You can destroy your user proxy by grid-proxy-destroy or glite-voms-proxy-destroy.
Line: 58 to 52
  This means getting a so called job proxy. Commands are:
Changed:
<
<
> myproxy-init -d -n Here, you will be prompted for your grid password.
>
>
> myproxy-init -d -n Here, you will be prompted for your grid password.
  Running myproxy-init is necessary when you are running long-term jobs. Having a job proxy ensures that your jobs still will be authenticated on the Grid, even though your user proxy (used to perform interactive Grid manupulations) may have had expired. You can get information on your job proxy by myproxy-info -d. You can destroy your job proxy by myproxy-destroy -d. The default (and maximum) lifetime of a job proxy is 168 hours.
Line: 88 to 82
  Requirements = ( Member("AFS", other.GlueHostApplicationSoftwareRunTimeEnvironment) && other.GlueCEPolicyMaxWallClockTime>=2160 &&
Changed:
<
<
other.GlueHostMainMemoryRAMSize>=512
>
>
other.GlueHostMainMemoryRAMSize>=512 && RegExp ("kfki.hu", other.GlueCEUniqueID);
  );

]

Line: 100 to 95
 
Executable
This variable specifies the executable file of your job.
StdOutput
The StdOut of your program shall be written into this file.
StdError
The StdError of your program shall be written into this file.
Changed:
<
<
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MegaBytes). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or http (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MegaBytes). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEPolicyMaxWallClockTime>=running_time means the requirement for such queues, where the job execution time limit is larger than the specified running_time (in minutes). The requirement other.GlueHostMainMemoryRAMSize>=memory means requirement for such execution nodes, which have larger memory than the specified memory amount (in MegaBytes). A requirement of the form other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid" would mean the requirement that the job should be sent to the computing element (queue) grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid.
>
>
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MegaBytes). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or http (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MegaBytes). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEPolicyMaxWallClockTime>=running_time means the requirement for such queues, where the job execution time limit is larger than the specified running_time (in minutes). The requirement other.GlueHostMainMemoryRAMSize>=memory means requirement for such execution nodes, which have larger memory than the specified memory amount (in MegaBytes). A requirement of the form RegExp("kfki.hu", other.GlueCEUniqueID) means, that the job should be sent to a KFKI computing element (either grid107.kfki.hu or grid109.kfki.hu).
  Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:
> glite-wms-job-list-match  --vo your_vo  testjob.jdl
Changed:
<
<
This will return a list of Grid queues (computing elements), which are capable of executing your job.
>
>

This will return a list of Grid queues (computing elements), which are capable of executing your job.

  The job can be submitted by the command:
> glite-wms-job-submit  --vo your_vo  -a  testjob.jdl
Changed:
<
<
This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.
>
>

This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.

  The status of the job can be viewed by:
> glite-wms-job-status  jobID
Changed:
<
<
This will return the current status of your job.
>
>

This will return the current status of your job.

  If your job has failed to be ran by the Grid system, the logging may be retrieved by:
> glite-wms-job-logging-info  jobID
Changed:
<
<
This will return the logging info on your job. A convenient way to find out failure reasons is:
>
>

This will return the logging info on your job. A convenient way to find out failure reasons is:

 > glite-wms-job-logging-info -v 3 jobID | grep "reason" | uniq
Changed:
<
<
This will return all the available most detailed logging info on your job, and shall print lines containing the string "reason", furthermore shall suppress multiple printing of consecutive identical lines.
>
>

This will return all the available most detailed logging info on your job, and shall print lines containing the string "reason", furthermore shall suppress multiple printing of consecutive identical lines.

  If your job has properly finished, you can retrieve the outputs by the command:
> glite-wms-job-output  jobID
Changed:
<
<
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID. It is also possible to specify some other directory name by the --dir switch.
>
>
 
Changed:
<
<
For further information, look at the man pages of the above commands, and maybe also to the man pages of other glite-wms- commands. For further references on simple job submission, see https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf. Also a complete description of the JDL language is available there.
>
>
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID. It is also possible to specify some other directory name by the --dir switch.
 
Added:
>
>
For further information, look at the man pages of the above commands, and maybe also to the man pages of other glite-wms- commands. For further references on simple job submission, see https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf. Also a complete description of the JDL language is available there.
  -- AndrasLaszlo - 10 Jul 2009 \ No newline at end of file

Revision 112009-07-10 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 133 to 133
  For further information, look at the man pages of the above commands, and maybe also to the man pages of other glite-wms- commands. For further references on simple job submission, see https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf. Also a complete description of the JDL language is available there.
Changed:
<
<
-- AndrasLaszlo - 4 Nov 2008
>
>
-- AndrasLaszlo - 10 Jul 2009
 \ No newline at end of file

Revision 102009-07-03 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 24 to 24
  setenv GLITE_USER_HOME /some_directory/glite (for tcsh).
Added:
>
>
By default, on every User Interface machine, the Grid Certificates of the trusted sites are located under the /etc/grid-security/certificates directory. If they are located, for whatever reason, under an alternative location, it can be told to the Grid applications by setting the following environmental variables:

export X509_CERT_DIR=/some_directory/certificates (for bash), or

setenv X509_CERT_DIR /some_directory/certificates (for tcsh).

 To customize the Grid behavior, the environmental variable GLITE_WMSUI_CONFIG_VAR can be used to point to a configuration file, like:

export GLITE_WMSUI_CONFIG_VAR=/some_directory/glite_wms.conf (for bash), or

setenv GLITE_WMSUI_CONFIG_VAR /some_directory/glite_wms.conf (for tcsh).

Changed:
<
<
The default version of the file glite_wms.conf may be obtained from /opt/glite/etc//glite_wms.conf on any UI machine. (However, the prefix /opt may be different on various platforms, if non-standard installation directory is used for GLite.)
>
>
The default version of the file glite_wms.conf may be obtained from /opt/glite/etc/glite_wms.conf on any UI machine. (However, the prefix /opt may be different on various platforms, if non-standard installation directory is used for GLite.)
 

Log onto the Grid (get authenticated on the Grid)

Revision 92008-11-21 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 32 to 30
  setenv GLITE_WMSUI_CONFIG_VAR /some_directory/glite_wms.conf (for tcsh).
Changed:
<
<
The default version of the file glite_wms.conf may be obtained from /opt/glite/etc//glite_wms.conf on any UI machine. (However, the prefix /opt may be different on various platforms, if non-standard installation directory is used for GLite.)
>
>
The default version of the file glite_wms.conf may be obtained from /opt/glite/etc//glite_wms.conf on any UI machine. (However, the prefix /opt may be different on various platforms, if non-standard installation directory is used for GLite.)
 

Log onto the Grid (get authenticated on the Grid)

Line: 42 to 40
  > grid-proxy-init  -valid 04:00 This is the same, but the authentication will expire in 4 hours. The default (and maximum) lifetime is 12 hours.
Changed:
<
<
If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:
>
>
If you are member of more then one VO, you can choose between them by using the glite-voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:
 
Changed:
<
<
> voms-proxy-init  -voms hungrid Or:
>
>
> glite-voms-proxy-init -voms hungrid Or:
 
Changed:
<
<
> voms-proxy-init  -voms hungrid  -valid 04:00

To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info  -all. You can destroy your user proxy by grid-proxy-destroy or voms-proxy-destroy.

>
>
> glite-voms-proxy-init -voms hungrid -valid 04:00
 
Added:
>
>
To get information on your user proxy, you can use the commands grid-proxy-info or glite-voms-proxy-info -all. You can destroy your user proxy by grid-proxy-destroy or glite-voms-proxy-destroy.
 

Get your jobs authenticated on the Grid

Line: 105 to 101
 Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:
> glite-wms-job-list-match  --vo your_vo  testjob.jdl
Changed:
<
<
This will return a list of Grid queues (computing elements), which are capable of executing your job.
>
>
This will return a list of Grid queues (computing elements), which are capable of executing your job.
  The job can be submitted by the command:
> glite-wms-job-submit  --vo your_vo  -a  testjob.jdl
Changed:
<
<
This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.
>
>
This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.
  The status of the job can be viewed by:
> glite-wms-job-status  jobID
Changed:
<
<
This will return the current status of your job.
>
>
This will return the current status of your job.
  If your job has failed to be ran by the Grid system, the logging may be retrieved by:
> glite-wms-job-logging-info  jobID
Changed:
<
<
This will return the logging info on your job. A convenient way to find out failure reasons is:
>
>
This will return the logging info on your job. A convenient way to find out failure reasons is:
 > glite-wms-job-logging-info -v 3 jobID | grep "reason" | uniq
Changed:
<
<
This will return all the available most detailed logging info on your job, and shall print lines containing the string "reason", furthermore shall suppress multiple printing of consecutive identical lines.
>
>
This will return all the available most detailed logging info on your job, and shall print lines containing the string "reason", furthermore shall suppress multiple printing of consecutive identical lines.
  If your job has properly finished, you can retrieve the outputs by the command:
> glite-wms-job-output  jobID
Changed:
<
<
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID. It is also possible to specify some other directory name by the --dir switch.
>
>
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID. It is also possible to specify some other directory name by the --dir switch.
  For further information, look at the man pages of the above commands, and maybe also to the man pages of other glite-wms- commands. For further references on simple job submission, see https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf. Also a complete description of the JDL language is available there.

Revision 82008-11-04 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 20 to 20
  setenv X509_USER_KEY /some_directory/globus/userkey.pem (for tcsh).
Changed:
<
<
To customize the Grid behavior, the environmental variable EDG_WL_UI_CONFIG_VAR can be used to point to a configuration file file, like:
>
>
By default, a directory called .glite should be present in your home directory. If, for whatever reason you wish to relocate this directory, you should tell this to the Grid system using the following environmental variables:
 
Changed:
<
<
export EDG_WL_UI_CONFIG_VAR=/some_directory/edg.conf (for bash), or
>
>
export GLITE_USER_HOME=/some_directory/glite (for bash), or
 
Changed:
<
<
setenv EDG_WL_UI_CONFIG_VAR /some_directory/edg.conf (for tcsh).
>
>
setenv GLITE_USER_HOME /some_directory/glite (for tcsh).
 
Changed:
<
<
The default version of the file edg.conf may be obtained from /opt/edg/etc/edg_wl_ui_cmd_var.conf on any UI machine.
>
>
To customize the Grid behavior, the environmental variable GLITE_WMSUI_CONFIG_VAR can be used to point to a configuration file, like:

export GLITE_WMSUI_CONFIG_VAR=/some_directory/glite_wms.conf (for bash), or

setenv GLITE_WMSUI_CONFIG_VAR /some_directory/glite_wms.conf (for tcsh).

The default version of the file glite_wms.conf may be obtained from /opt/glite/etc//glite_wms.conf on any UI machine. (However, the prefix /opt may be different on various platforms, if non-standard installation directory is used for GLite.)

 

Log onto the Grid (get authenticated on the Grid)

Line: 34 to 40
  > grid-proxy-init Here, you will be prompted for your grid password. Or:
Changed:
<
<
> grid-proxy-init  -valid 4:00 This is the same, but the authentication will expire in 4 hours. The default (and maximum) lifetime is 12 hours.
>
>
> grid-proxy-init  -valid 04:00 This is the same, but the authentication will expire in 4 hours. The default (and maximum) lifetime is 12 hours.
  If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:

> voms-proxy-init  -voms hungrid Or:

Changed:
<
<
> voms-proxy-init  -voms hungrid  -valid 4:00
>
>
> voms-proxy-init  -voms hungrid  -valid 04:00
  To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info  -all. You can destroy your user proxy by grid-proxy-destroy or voms-proxy-destroy.
Line: 49 to 55
  This means getting a so called job proxy. Commands are:
Changed:
<
<
> myproxy-init Here, you will be prompted for your grid password, and to specify an additional password (for extra protection) attached to your so called job proxy, to be created.

> myproxy-init  -n This is the same, but you won't be asked to specify an additional password for protecting your job proxy.

>
>
> myproxy-init  -d  -n Here, you will be prompted for your grid password.
 
Changed:
<
<
Running myproxy-init is necessary when you are running long-term jobs. Having a job proxy ensures that your jobs still will be authenticated on the Grid, even though your user proxy (used to perform interactive Grid manupulations) may have had expired. You can get information on your job proxy by myproxy-info. You can destroy your job proxy by myproxy-destroy. The default (and maximum) lifetime of a job proxy is 168 hours.
>
>
Running myproxy-init is necessary when you are running long-term jobs. Having a job proxy ensures that your jobs still will be authenticated on the Grid, even though your user proxy (used to perform interactive Grid manupulations) may have had expired. You can get information on your job proxy by myproxy-info  -d. You can destroy your job proxy by myproxy-destroy  -d. The default (and maximum) lifetime of a job proxy is 168 hours.
  Note: If you don't get a job proxy, you may not be able to retrieve your job outputs for long-term jobs!

Prepare and submit your job

Changed:
<
<
The programme which you want to run on the Grid is called a job. These consist of some executable(s) and some input(s), which can be submitted to the Grid system. The result shall be some output(s), which can be retrieved after your job has finished.
>
>
The program which you want to run on the Grid is called a job. These consist of some executable(s) and some input(s), which can be submitted to the Grid system. The result shall be some output(s) and error(s), which can be retrieved after your job has finished.
  The job specifications are described for the Grid system by the so called Job Description Language (JDL). For each of your jobs, you should prepare a JDL file. An example for a typical simple JDL file content may be:
Line: 71 to 75
  Executable = "testjob.sh";
Changed:
<
<
StdOutput = "testjob.stdout";
>
>
StdOutput = "testjob.stdoutanderror";
 
Changed:
<
<
StdError = "testjob.stdout";
>
>
StdError = "testjob.stdoutanderror";
  InputSandbox = {"testjob.sh", "inputfile.dat"};
Changed:
<
<
OutputSandbox = {"testjob.stdout", "outputfile.dat"};
>
>
OutputSandbox = {"testjob.stdoutanderror", "outputfile.dat"};
  Requirements = ( Member("AFS", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
Line: 100 to 104
  Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:
Changed:
<
<
> edg-job-list-match -vo your_vo testjob.jdl
>
>
> glite-wms-job-list-match --vo your_vo testjob.jdl
  This will return a list of Grid queues (computing elements), which are capable of executing your job.

The job can be submitted by the command:

Changed:
<
<
> edg-job-submit -vo your_vo testjob.jdl
>
>
> glite-wms-job-submit --vo your_vo -a testjob.jdl
  This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.

The status of the job can be viewed by:

Changed:
<
<
> edg-job-get-status jobID
>
>
> glite-wms-job-status jobID
  This will return the current status of your job.

If your job has failed to be ran by the Grid system, the logging may be retrieved by:

Changed:
<
<
> edg-job-get-logging-info jobID
>
>
> glite-wms-job-logging-info jobID
  This will return the logging info on your job. A convenient way to find out failure reasons is:
Changed:
<
<
> edg-job-get-logging-info -v 2 jobID > log
> grep "reason" log | uniq
>
>
> glite-wms-job-logging-info -v 3 jobID | grep "reason" | uniq
 
Changed:
<
<
This will return all the available logging info on your job (-v 2 switch), and shall write it into the file log. The second command line lists the unique lines of the file log, containing the string reason, which will tell the reasons for various actions of the Grid system.
>
>
This will return all the available most detailed logging info on your job, and shall print lines containing the string "reason", furthermore shall suppress multiple printing of consecutive identical lines.
  If your job has properly finished, you can retrieve the outputs by the command:
Changed:
<
<
> edg-job-get-output jobID
>
>
> glite-wms-job-output jobID
  This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID. It is also possible to specify some other directory name by the --dir switch.
Changed:
<
<
For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands. For further references on simple job submission, see https://edms.cern.ch/file/454439//LCG-2-UserGuide.html. Also a complete description of the JDL language is available there.
>
>
For further information, look at the man pages of the above commands, and maybe also to the man pages of other glite-wms- commands. For further references on simple job submission, see https://edms.cern.ch/file/722398//gLite-3-UserGuide.pdf. Also a complete description of the JDL language is available there.
 

Changed:
<
<
-- AndrasLaszlo - 17 Sep 2007
>
>
-- AndrasLaszlo - 4 Nov 2008

Revision 72008-04-06 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 34 to 34
  > grid-proxy-init Here, you will be prompted for your grid password. Or:
Changed:
<
<
> grid-proxy-init  -valid 4:00 This is the same, but the authentication will expire in 4 hours (default lifetime is 12 hours).
>
>
> grid-proxy-init  -valid 4:00 This is the same, but the authentication will expire in 4 hours. The default (and maximum) lifetime is 12 hours.
  If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:
Line: 42 to 42
  > voms-proxy-init  -voms hungrid  -valid 4:00
Changed:
<
<
To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info  -all.
>
>
To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info  -all. You can destroy your user proxy by grid-proxy-destroy or voms-proxy-destroy.
 

Get your jobs authenticated on the Grid

Line: 53 to 53
  > myproxy-init  -n This is the same, but you won't be asked to specify an additional password for protecting your job proxy.
Changed:
<
<
Running myproxy-init is necessary when you are running long-term jobs. Having a job proxy ensures that your jobs still will be authenticated on the Grid, even though your user proxy (used to perform interactive Grid manupulations) may have had expired. You can get information on your job proxy by myproxy-info. You can destroy your job proxy by myproxy-destroy. The default lifetime of a job proxy is 168 hours.
>
>
Running myproxy-init is necessary when you are running long-term jobs. Having a job proxy ensures that your jobs still will be authenticated on the Grid, even though your user proxy (used to perform interactive Grid manupulations) may have had expired. You can get information on your job proxy by myproxy-info. You can destroy your job proxy by myproxy-destroy. The default (and maximum) lifetime of a job proxy is 168 hours.
  Note: If you don't get a job proxy, you may not be able to retrieve your job outputs for long-term jobs!
Line: 131 to 131
 
> edg-job-get-output  jobID
Changed:
<
<
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID.
>
>
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID. It is also possible to specify some other directory name by the --dir switch.
  For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands. For further references on simple job submission, see https://edms.cern.ch/file/454439//LCG-2-UserGuide.html. Also a complete description of the JDL language is available there.

Revision 62007-12-04 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 10 to 10
  After you have logged onto a UI machine, you are able to submit commands to the Grid.
Added:
>
>
You should have your Grid user key files usercert.pem and userkey.pem under ~/.globus directory. If they are located, for whatever reason, under an alternative location, it can be told to the Grid applications by setting the following environmental variables, like:

export X509_USER_CER=/some_directory/globus/usercert.pem

export X509_USER_KEY=/some_directory/globus/userkey.pem (for bash), or

setenv X509_USER_CER /some_directory/globus/usercert.pem

setenv X509_USER_KEY /some_directory/globus/userkey.pem (for tcsh).

To customize the Grid behavior, the environmental variable EDG_WL_UI_CONFIG_VAR can be used to point to a configuration file file, like:

export EDG_WL_UI_CONFIG_VAR=/some_directory/edg.conf (for bash), or

setenv EDG_WL_UI_CONFIG_VAR /some_directory/edg.conf (for tcsh).

The default version of the file edg.conf may be obtained from /opt/edg/etc/edg_wl_ui_cmd_var.conf on any UI machine.

 

Log onto the Grid (get authenticated on the Grid)

Revision 52007-12-04 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 64 to 64
  Requirements = ( Member("AFS", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
Changed:
<
<
other.GlueCEPolicyMaxWallClockTime>=2160
>
>
other.GlueCEPolicyMaxWallClockTime>=2160 && other.GlueHostMainMemoryRAMSize>=512
  );

]

Line: 76 to 77
 
Executable
This variable specifies the executable file of your job.
StdOutput
The StdOut of your program shall be written into this file.
StdError
The StdError of your program shall be written into this file.
Changed:
<
<
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MB). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or http (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MB). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEPolicyMaxWallClockTime>=maximal_running_time means the requirement for such queues, where the job execution time limit is larger than the specified maximal_running_time (in minutes). A requirement of the form other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid" would mean the requirement that the job should be sent to the computing element (queue) grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid.
>
>
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MegaBytes). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or http (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MegaBytes). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEPolicyMaxWallClockTime>=running_time means the requirement for such queues, where the job execution time limit is larger than the specified running_time (in minutes). The requirement other.GlueHostMainMemoryRAMSize>=memory means requirement for such execution nodes, which have larger memory than the specified memory amount (in MegaBytes). A requirement of the form other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid" would mean the requirement that the job should be sent to the computing element (queue) grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid.
  Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:

Revision 42007-11-28 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 19 to 19
  > grid-proxy-init  -valid 4:00 This is the same, but the authentication will expire in 4 hours (default lifetime is 12 hours).
Changed:
<
<
If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead of grid-proxy-init. E.g.:
>
>
If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead of grid-proxy-init command. E.g.:
  > voms-proxy-init  -voms hungrid Or:

> voms-proxy-init  -voms hungrid  -valid 4:00

Changed:
<
<
To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info.
>
>
To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info  -all.
 

Get your jobs authenticated on the Grid

Line: 64 to 64
  Requirements = ( Member("AFS", other.GlueHostApplicationSoftwareRunTimeEnvironment) &&
Changed:
<
<
other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid"
>
>
other.GlueCEPolicyMaxWallClockTime>=2160
  );

]

Line: 76 to 76
 
Executable
This variable specifies the executable file of your job.
StdOutput
The StdOut of your program shall be written into this file.
StdError
The StdError of your program shall be written into this file.
Changed:
<
<
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MB). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or the internet (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MB). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid" means the requirement that the job should be sent to the computing element (queue) grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid.
>
>
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MB). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or http (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MB). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEPolicyMaxWallClockTime>=maximal_running_time means the requirement for such queues, where the job execution time limit is larger than the specified maximal_running_time (in minutes). A requirement of the form other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid" would mean the requirement that the job should be sent to the computing element (queue) grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid.
  Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:
Changed:
<
<
> edg-list-match -vo your_vo testjob.jdl
>
>
> edg-job-list-match -vo your_vo testjob.jdl
 
Changed:
<
<
This will return a list of Grid queues (computing elements).
>
>
This will return a list of Grid queues (computing elements), which are capable of executing your job.
  The job can be submitted by the command:
Line: 115 to 115
  This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID.
Changed:
<
<
For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands. For further references on simple job submission, see https://edms.cern.ch/file/454439//LCG-2-UserGuide.html.
>
>
For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands. For further references on simple job submission, see https://edms.cern.ch/file/454439//LCG-2-UserGuide.html. Also a complete description of the JDL language is available there.
 

Revision 32007-09-19 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Revision 22007-09-17 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

Line: 6 to 6
 On this page, we intend to provide a brief introduction on simple job submission onto the Grid. One can also find a brief introductory material on the Grid homepage of RMKI.
Changed:
<
<
  • Log onto a User Interface (UI) machine.
>
>

Log onto a User Interface (UI) machine

  After you have logged onto a UI machine, you are able to submit commands to the Grid.
Changed:
<
<
  • Log onto the Grid (get authenticated on the Grid).
>
>

Log onto the Grid (get authenticated on the Grid)

  This means getting a so called user proxy. Commands are:
Line: 19 to 19
  > grid-proxy-init  -valid 4:00 This is the same, but the authentication will expire in 4 hours (default lifetime is 12 hours).
Changed:
<
<
If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead. E.g.:
>
>
If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead of grid-proxy-init. E.g.:
  > voms-proxy-init  -voms hungrid Or:
Line: 28 to 28
 To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info.
Changed:
<
<
  • Get your jobs authenticated on the Grid.
>
>

Get your jobs authenticated on the Grid

  This means getting a so called job proxy. Commands are:
Line: 41 to 41
 Note: If you don't get a job proxy, you may not be able to retrieve your job outputs for long-term jobs!
Changed:
<
<
  • Prepare and submit your job.
>
>

Prepare and submit your job

  The programme which you want to run on the Grid is called a job. These consist of some executable(s) and some input(s), which can be submitted to the Grid system. The result shall be some output(s), which can be retrieved after your job has finished.
Line: 115 to 115
  This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID.
Changed:
<
<
For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands.
>
>
For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands. For further references on simple job submission, see https://edms.cern.ch/file/454439//LCG-2-UserGuide.html.

-- AndrasLaszlo - 17 Sep 2007

Revision 12007-09-16 - AndrasLaszlo

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

How to submit simple jobs onto the Grid

On this page, we intend to provide a brief introduction on simple job submission onto the Grid. One can also find a brief introductory material on the Grid homepage of RMKI.

  • Log onto a User Interface (UI) machine.

After you have logged onto a UI machine, you are able to submit commands to the Grid.

  • Log onto the Grid (get authenticated on the Grid).

This means getting a so called user proxy. Commands are:

> grid-proxy-init Here, you will be prompted for your grid password. Or:

> grid-proxy-init  -valid 4:00 This is the same, but the authentication will expire in 4 hours (default lifetime is 12 hours).

If you are member of more then one VO, you can choose between them by using the voms-proxy-init for logging in, instead. E.g.:

> voms-proxy-init  -voms hungrid Or:

> voms-proxy-init  -voms hungrid  -valid 4:00

To get information on your user proxy, you can use the commands grid-proxy-info or voms-proxy-info.

  • Get your jobs authenticated on the Grid.

This means getting a so called job proxy. Commands are:

> myproxy-init Here, you will be prompted for your grid password, and to specify an additional password (for extra protection) attached to your so called job proxy, to be created.

> myproxy-init  -n This is the same, but you won't be asked to specify an additional password for protecting your job proxy.

Running myproxy-init is necessary when you are running long-term jobs. Having a job proxy ensures that your jobs still will be authenticated on the Grid, even though your user proxy (used to perform interactive Grid manupulations) may have had expired. You can get information on your job proxy by myproxy-info. You can destroy your job proxy by myproxy-destroy. The default lifetime of a job proxy is 168 hours.

Note: If you don't get a job proxy, you may not be able to retrieve your job outputs for long-term jobs!

  • Prepare and submit your job.

The programme which you want to run on the Grid is called a job. These consist of some executable(s) and some input(s), which can be submitted to the Grid system. The result shall be some output(s), which can be retrieved after your job has finished.

The job specifications are described for the Grid system by the so called Job Description Language (JDL). For each of your jobs, you should prepare a JDL file. An example for a typical simple JDL file content may be:

[

JobType = "Normal"

Executable = "testjob.sh";

StdOutput = "testjob.stdout";

StdError = "testjob.stdout";

InputSandbox = {"testjob.sh", "inputfile.dat"};

OutputSandbox = {"testjob.stdout", "outputfile.dat"};

Requirements = (
                 Member("AFS", other.GlueHostApplicationSoftwareRunTimeEnvironment) && 
                 other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid"
                     );

]

The meaning of the above variables are:

JobType
This optional variable describes whether your job is a normal job ("Normal"), or interactive ("Interactive"). If your job is interactive, the StdIn, StdOut and StdError shall be connected to the terminal, from where you submitted the job, so you are able to communicate with the job during the running time. If unspecified, defaults to "Normal".
Executable
This variable specifies the executable file of your job.
StdOutput
The StdOut of your program shall be written into this file.
StdError
The StdError of your program shall be written into this file.
InputSandbox
This is a list of files, which are sent to the system as the components of your job. Typically the executable of your program, and some supplementary files. The size of the files, sent via the InputSandbox, should be small (<10MB). Large files (as large input data files) should be communicated to the job by other ways, e.g. via AFS, NFS, or the internet (using e.g. wget), or via the Grid Storage System.
OutputSandbox
This is a list of files, which are retrieved after the job has finished. Typically the file containing the StdOut / StdError and some output files. The size of the files, retrieved via OutputSandbox, should be small (<100MB). Large files (as large output data files) should be transfered by other means, e.g. via the Grid Storage System.
Requirements
This optional variable may be a logical expression, specifying requirements for site or the node, where the job is going to be executed. Member("Some_software", other.GlueHostApplicationSoftwareRunTimeEnvironment) means the requirement of the software Some_software on the target node. other.GlueCEUniqueID=="grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid" means the requirement that the job should be sent to the computing element (queue) grid109.kfki.hu:2119/jobmanager-lcgpbs-hungrid.

Once you prepared the JDL file, you can look for the available queues, which are capable of running your job, by the command:

> edg-list-match  -vo your_vo  testjob.jdl
This will return a list of Grid queues (computing elements).

The job can be submitted by the command:

> edg-job-submit  -vo your_vo  testjob.jdl
This will return a sURL address, which is a unique identifier of your job, which shall be denoted by jobID in the followings.

The status of the job can be viewed by:

> edg-job-get-status  jobID
This will return the current status of your job.

If your job has failed to be ran by the Grid system, the logging may be retrieved by:

> edg-job-get-logging-info  jobID
This will return the logging info on your job. A convenient way to find out failure reasons is:
> edg-job-get-logging-info  -v 2  jobID  > log
> grep  "reason"  log  |  uniq
This will return all the available logging info on your job (-v 2 switch), and shall write it into the file log. The second command line lists the unique lines of the file log, containing the string reason, which will tell the reasons for various actions of the Grid system.

If your job has properly finished, you can retrieve the outputs by the command:

> edg-job-get-output jobID
This will retrieve the content of the OutputSandbox into the directory /tmp/jobOutput/yourusername_jobID.

For further information, look at the man pages of the above commands, and maybe also to the man pages of other edg- commands.

 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback