Difference: LHCComputingGrid (6 vs. 7)

Revision 72006-03-06 - AndrasLaszlo

Line: 1 to 1
 Getting access to LHC Computing Grid. One can find a brief introductory material at the RMKI's getting started page, where you can find information on how to get access to LCG. There are also simple examples on that page.

Some more practical information on running a typical job.

Line: 21 to 21
 
  • Running a job:
Changed:
<
<
A "Hello World!" example can be found at RMKI's getting started page. Instead of a "Hello World!" example, we present here a framework, with which one can send jobs in mass to the LCG: see the attached tarball called submit.tar. You can adjust these shell script wrappers to your needs. This assumes that your jobs are placed in a directory scheme like in the attached example skeleton.tar. This latter is a framework for simple jobs, which are ran on a simple computers (contains an automated Makefile and an automated starter shell script).
>
>
A "Hello World!" example can be found at RMKI's getting started page. Instead of a "Hello World!" example, we present here a framework, with which one can send jobs in mass to the LCG: see the attached tarball called submit.tar.gz. You can adjust these shell script wrappers to your needs. This assumes that your jobs are placed in a directory scheme like in the attached example skeleton.tar.gz. This latter is a framework for simple jobs, which are ran on a simple computers (contains an automated Makefile and an automated starter shell script).
 
  • Specifying system requiremets:
Line: 35 to 35
 
  • Using the storage elements of the LCG:
Added:
>
>
The files are stored on Storage Elements on the LCG. Usually these have different physical file administration schemes. Therefore, a logical layer was built on top of them, which makes the file access uniform from the user point of view. To use this facility , use the environmental variable LCG_CATALOG_TYPE:
 
Added:
>
>
export LCG_CATALOG_TYPE = lfc (for bash), or setenv LCG_CATALOG_TYPE lfc (for tcsh).
 
Changed:
<
<
-- AndrasLaszlo - 03 Mar 2006
>
>
The Logical File Name (lfn) is resolved by the Logical File Catalogue (lfc) server. You have to specify this:
 
Added:
>
>
export LFC_HOST = lfc-cms-test.cern.ch (for bash), or setenv LFC_HOST lfc-cms-test.cern.ch (for tcsh).
 
Added:
>
>
Now you can refer to files with their logical file names, which are site-independent. The logical file names have to be unique. A very convenient way to generate them is to imitate the UNIX type file naming scheme. With the command lfc-mkdir one can make logical directories, like

> lfc-mkdir -m 755 /grid/cms/user_name/some_directory (Here, the -m xxx option is optional, this sets the permissions of the directory. All logical file names begin with /grid/virtual_organization. If you are a new user, the directory /grid/virtual_organization/user_name will not exist initially: you have to create it first, if you want an own directory. It is recommended to restrict write permissions like in the example.)

There are various commands to manage logical file names, all beginning with lfc-, like lfc-chmod, lfc-getacl, lfc-setacl, lfc-chown, lfc-ln, lfc-rename, lfc-ls, lfc-rm, lfc-mkdir etc. Refer to the man pages of these commands, but the usage of these should be self-explanatory.

As the given logical directory is ready, one can copy files into it. If you want to store a file in the LCG storage infrastructure permanently, copying a file is not enough: you have to register the file in the logical file catalogue. It is done like:

> lcg-cr --vo cms -d grid100.kfki.hu -l lfn:/grid/cms/alaszlo/destionation/testfile.txt \
                                                    file:/afs/kfki.hu/home/alaszlo/source/testfile.txt
(Here, the -d destination_Storage_Element is optional, and is used to force the destination Storage Element to be destination_Storage_Element; otherwise the file may be copied to a Storage Element anywhere in the World.) This command will copy and register your file onto LCG storage system. Once your file is there, you can get a copy of it with lcg-cp e.g. as an input for a job:

> lcg-cp --vo cms lfn:/grid/cms/alaszlo/some_directory/test_file.txt file:$PWD/test_file.txt
(This command stages out the file in question onto a local --or AFS-- disc area, pointed at by $PWD, i.e. into your current directory.)

There are various commands to manage the stored files (e.g. unregister and delete them, or create replicas), all beginning with lcg-, like lcg-rep, lcg-aa, lcg-rf, lcg-cp, lcg-cr, lcg-la, lcg-uf, lcg-del, lcg-lg, lcg-lr, lcg-gt, lcg-ra etc. Refer to the man peges of these commands, but most of them are self-explanatory.

Unfortunately, one cannot copy and register complete directory trees. Therefore, I wrote a shell script wrapper to do this:

> lcgcr.sh source_directory destination_directory_lfn (You can find this here: lcgcr.sh.)

Note: unfortunately, I encountered that the lcg-cr action often fails (this should be remedied in future). Therefore, it is convenient to put an lcg-cr command into a checker loop:

RESULT = 1
while (( $RESULT != 0 )) ; do
    lcg-cr --vo cms -d grid100.kfki.hu \
        -l lfn:/grid/cms/alaszlo/destintation/test_file.txt \
            file:/afs/kfki.hu/home/alaszlo/source/test_file.txt
    RESULT = $?
    if (( $RESULT != 0 )) ; then sleep 1m ; fi
done

This precaution may also be recommended for lcg-cp.

-- AndrasLaszlo - 06 Mar 2006

META FILEATTACHMENT attr="" autoattached="1" comment="Framework for a simple job" date="1141665617" name="skeleton.tar.gz" path="skeleton.tar.gz" size="4071" user="Main.AndrasLaszlo" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Framework for a simple job" date="1141385472" name="skeleton.tar" path="skeleton.tar" size="30720" user="Main.AndrasLaszlo" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="A simple LCG mass submitter framework" date="1141665685" name="submit.tar.gz" path="submit.tar.gz" size="7791" user="Main.AndrasLaszlo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Copy and register complete directory trees into Logical File Catalogue" date="1141665733" name="lcgcr.sh" path="lcgcr.sh" size="3866" user="Main.AndrasLaszlo" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="A simple LCG mass submitter framework" date="1141385536" name="submit.tar" path="submit.tar" size="61440" user="Main.AndrasLaszlo" version="1"
META TOPICMOVED by="AndrasLaszlo" date="1141225754" from="CMS.WebTopicCreator" to="CMS.LHCComputingGrid"
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback