Difference: MassSe (7 vs. 8)

Revision 82008-11-21 - AndrasLaszlo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Mass file handling on the Grid (the GSTREAM library)

Line: 6 to 6
 

The GSTREAM library for read/write C++ streams to Storage Elements

Changed:
<
<		As one does generally not want to always stage out the data files from SE-s onto a local disk by hand, and then process it, it is recommended to have read/write streams. The C++ library GSTREAM implements such a library. (gstream, igstream, ogstream stream classes, like the usual C++ STL fstream, ifstream, ofstream file input/output stream classes; the letter 'g' standing for 'grid'.) It does nothing else, but treats the file as a normal file, unless its name begins with the string /grid/. In this case, it stages out the datafile in question onto a local (or AFS) area, and then treats the local file as a normal file. Sooner or later this solution has to be replaced with a GFAL based C++ library.
>
>		As one does generally not want to always stage out the data files from SE-s onto a local disk by hand, and then process it, it is recommended to have read/write streams. The C++ library GSTREAM implements such a library. ( gstream, igstream, ogstream stream classes, like the usual C++ STL fstream, ifstream, ofstream file input/output stream classes; the letter 'g' standing for 'grid'.) It does nothing else, but treats the file as a normal file, unless its name begins with the string /grid/. In this case, it stages out the datafile in question onto a local (or AFS) area, and then treats the local file as a normal file. Sooner or later this solution has to be replaced with a GFAL based C++ library.
  One commonly faces the problem that the file not only has to be processed, but it also has to be passed through a filter program. Therefore I also wrote pipe streams for grid storage (igpstream, ogpstream), which are based on the ipstream and opstream classes of the library at http://pstreams.sourceforge.net (note the LGPL license!).
Line: 54 to 53
  Before usage, one has to export the following environmental variables:
Changed:
<
<
export LCG_GFAL_VO=your_vo
(for bash), or 
setenv LCG_GFAL_VO your_vo
(for tcsh), and
>
>
export LCG_GFAL_VO=your_vo
(for bash), or 
setenv LCG_GFAL_VO your_vo
(for tcsh), and
 
Changed:
<
<
export DEST=your_favourite_storage_element1,your_storage_element2,...
(for bash), or 
setenv DEST your_favourite_storage_element1,your_storage_element2,...
(for tcsh).
>
>
export DEST=your_favourite_storage_element1,your_storage_element2,...
(for bash), or 
setenv DEST your_favourite_storage_element1,your_storage_element2,...
(for tcsh).
  Setting the environmental variable TMPDIR is optional. This specifies the local (or AFS) directory, where the datafiles are staged out (therefore, it has to have large disk space!). E.g.:
Changed:
<
<
export TMPDIR=/tmp
(for bash), or 
setenv TMPDIR /tmp
(for tcsh).
>
>
export TMPDIR=/tmp
(for bash), or 
setenv TMPDIR /tmp
(for tcsh).
  If not specified, the current working directory ($PWD) is used, as this is recommended for grid jobs (the working nodes have large disk spaces).
Line: 77 to 72
 
> lcg-cr.sh  /home/user_name/source  /grid/your_vo/user_name/destination  your_favourite_se  your_vo  > runplan.dat
> runplan.sh  runplan.dat
Changed:
<
<		 Here source may be large directory tree structure.
>
>		 Here source may be large directory tree structure.
  
> lcg-rep.sh  /grid/your_vo/user_name/source  destination_se  your_vo  > runplan.dat
> runplan.sh  runplan.dat
Changed:
<
<		 Here source may be large directory tree structure. destination_se is the SE, where you want the replica.
>
>		 Here source may be large directory tree structure. destination_se is the SE, where you want the replica.
  
> lcg-cp.sh  /grid/your_vo/user_name/source  /home/user_name/dest  your_vo  > runplan.dat
> runplan.sh  runplan.dat
Changed:
<
<		 Here source may be large directory tree structure.
>
>		 Here source may be large directory tree structure.
  
> lcg-del.sh  /grid/your_vo/user_name/target  target_se  your_vo  > runplan.dat
Line: 100 to 93
  The wrapper scripts may be improved in the future. We thank to Kálmán Kővári for this toolkit. There are also other useful scripts in the package. Please have a look a them.
Added:
>
>		Also the scripts gexists.sh, gput.sh, grep.sh, gget.sh, gdel.sh may be helpful to manipulate single files.
  -- AndrasLaszlo - 29 Jun 2008
Changed:
<
<
META FILEATTACHMENT attachment="gstream.tar.gz" attr="" comment="The GSTREAM library source." date="1224145996" name="gstream.tar.gz" path="gstream.tar.gz" size="18743" stream="gstream.tar.gz" user="Main.AndrasLaszlo" version="1"
>
>
META FILEATTACHMENT attachment="gstream.tar.gz" attr="" comment="The GSTREAM library source." date="1227279718" name="gstream.tar.gz" path="gstream.tar.gz" size="18850" stream="gstream.tar.gz" tmpFilename="/usr/tmp/CGItemp27205" user="AndrasLaszlo" version="4"
View topic | History: r13 < r12 < r11 < r10 | More topic actions...
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback