(r1) BasicSe < RmiGrid

RmiGrid Web>BasicSe (revision 1) (raw view)~~EditAttach~~
---++ Using the Storage Elements (_SE_) of the Grid


   * Command line tools.

The files are stored on Storage Elements on the Grid. These are computers with very large disk space (in the order of 100TB), located at different sites. Usually these have different physical file administration schemes. Therefore, a logical layer was built on top of them, which makes the file access uniform from the user point of view. To use this facility , use the environmental variable =LCG_CATALOG_TYPE=:

<verbatim>export LCG_CATALOG_TYPE=lfc</verbatim> (for _bash_), or
<verbatim>setenv LCG_CATALOG_TYPE lfc</verbatim> (for _tcsh_).

The Logical File Name (_lfn_) is resolved by the Logical File Catalogue (_lfc_) server. It can be thought of as the partitional table of a world-wide hard disk. You have to specify this database server, e.g.:

<verbatim>export LFC_HOST=grid155.kfki.hu</verbatim> (for _bash_), or
<verbatim>setenv LFC_HOST grid155.kfki.hu</verbatim> (for _tcsh_).

Now you can refer to files with their logical file names, which are site-independent. You do not have to care afterwords that to which site your files are actually written. The logical file names have to be unique.A very convenient way to generate them is to imitate the UNIX type file naming scheme. With the command =lfc-mkdir= one can make logical directories, like:

=> lfc-mkdir /grid/hungrid/user_name/some_directory= (All logical file names begin with =/grid/virtual_organization=. If you are a new user, the directory =/grid/virtual_organization/user_name= will not exist initially: you have to create it first, if you want an own directory.)

There are various commands to manage logical file names, all beginning with =lfc-=, like =lfc-chmod=, =lfc-getacl=, =lfc-setacl=, =lfc-chown=, =lfc-ln=, =lfc-rename=, =lfc-ls=, =lfc-rm=, =lfc-mkdir= etc. Refer to the =man= pages of these commands, but the usage of these should be self-explanatory, imitating their UNIX analogue.

As the given logical directory is ready, one can copy files into it. If you want to store a file in the Grid storage infrastructure permanently, copying a file onto a storage element is not enough: you have to register the file in the logical file catalogue. It is done like:

<verbatim>
> lcg-cr  --vo hungrid  -d grid143.kfki.hu  -l lfn:/grid/hungrid/user_name/destionation/test_file.txt  \
                                                                         file:/home/user_name/source/test_file.txt
</verbatim>
(Here, the =-d destination_Storage_Element= is optional, and is used to force the destination Storage Element to be =destination_Storage_Element=; otherwise the file may be copied to a Storage Element anywhere in the world.) This command will _copy and register your file_ onto the Grid storage system. Once your file is there, you can get a copy of it with =lcg-cp= e.g. as an input for a job:

<verbatim>
> lcg-cp  --vo hungrid  lfn:/grid/hungrid/user_name/source/test_file.txt  \
                                          file:/home/user_name/destination/test_file.txt
</verbatim>
This command gets a copy of your file on the storage system to some local area.

There are various commands to manage the stored files (e.g. unregister and delete them, or create replicas), all beginning with =lcg-=, like =lcg-rep=, =lcg-aa=, =lcg-rf=, =lcg-cp=, =lcg-cr=, =lcg-la=, =lcg-uf=, =lcg-del=, =lcg-lg=, =lcg-lr=, =lcg-gt=, =lcg-ra= etc. Refer to the =man= pages of these commands, but most of them are self-explanatory.

It is possible to spare always typeing =--vo your_vo=, by setting the environmental variable =LCG_GFAL_VO=:

<verbatim>export LCG_GFAL_VO=your_vo</verbatim> (for _bash_), or
<verbatim>setenv LCG_GFAL_VO your_vo</verbatim> (for _tcsh_).

_Note1_: unfortunately, I encountered that the =lcg-cr= action often fails (this may be remedied in future, or may be already solved). Therefore, it is convenient to put an =lcg-cr= command into a checker loop, like:

<verbatim>
#!/bin/sh
RESULT=1
TRIES=0
while (( $RESULT != 0 )) ; do
    lcg-cr --vo cms -d grid100.kfki.hu \
        -l lfn:/grid/cms/alaszlo/destintation/test_file.txt \
            file:/afs/kfki.hu/home/alaszlo/source/test_file.txt
    RESULT=$?
    TRIES=$(($TRIES+1))
    if (( $TRIES >= 5 )) ; then echo "lcg-cr failed. Giving up
after 5 tries." ; exit 1 ; fi
    if (( $RESULT != 0 )) ; then sleep 1m ; fi
done
</verbatim>

This precaution may also be recommended for =lcg-cp= or other =lcg-= commands.

_Note2_: Never restrict the _lfc_ permissions of a file or a directory in such a way, that the group (VO) does not have write permissions. There is a very simple reason for it. It is not really you, who does the actual copying onto a storage element: some program copies it on behalf of you, which is assigned some more or less random local user identifier, which is mapped to your Grid user identifier temporarily. This is a rather questionable way of manging user identifiers, but that is the way how it is implemented in the current version. This fact has rahter dangerous implications. If you restrict the permissions, such that the group does not have write permissions, then at an other occasion (when your Grid user identifier is mapped to another local user identifier), you cannot write your files. Meanwhile, an other user (who is accidentally given that particular local user identifier) may still have write permission to your file, despite of your precaution. As a consequence, the _lfc_ permissions should not be restricted in such a way that the group does not have write permissions. But this also has a consequence: practically anyone has write permissions
to your data in your group, plus you have write permissions to the data of others in the group. Let's hope that this conceptional mistake will be corrected soon.


   * Read/Write C routines (the GFAL).

The Grid File Access Library (GFAL) is a collection of read/write C routines. It is imitating the C file read/write routines. We refer to the =man= pages of =gfal=.
Topic revision: r1 - 2007-09-15 - AndrasLaszlo
RmiGrid
Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback