Storage Resource in GRID

In a Grid environment coexist heterogeneous storage resources. Basically, storage resources can be composed by disks, tapes or a combination of the two. The main logical entities of a storage resource are space and file. Most Grid applications involve the generation of large datasets, the consumption of large datasets, or both. In this scenario, the need to deal with reservation and scheduling of storage resources becomes fundamental.

Storage Reosource Manager [SRM]

Storage Resource Managers are middleware services whose function is to provide space allocation and file management of shared storage components. In that way, files are no longer permanent entities on the storage, but dynamical ones that can appear or disappear according to the user’s specification.
SRM services agree on a standard interface to hide storage characteristics and to allow interoperability. SRMs are implemented through the web service technology.
SRM is based on these concepts:

  • lifetime of a file (volatile with a fix amount of lifetime, durable or permanent).
  • file pinning (to ensure a file is not canceled during operation).
  • space pre-allocation (to ensure the request space is available for the whole life of the application since the beginning).
  • storage classes to identify different quality of storage resources.

StoRM

StoRM is a storage resource manager for disk based storage systems. It implements the SRM interface version 2.2. StoRM is designed to support guaranteed space reservation and direct access (native POSIX I/O call), as well as other standard libraries (like RFIO). StoRM takes advantage from high performance parallel file systems like GPFS (from IBM). In addition, standard POSIX file systems are supported (XFS from SGI and ext3). StoRM takes advantage of ACL support provided by the underlying file systems to implement the security models so it caters the interests of Economics and Finances as represented by the EGRID project.
A modular architecture decouples StoRM logic from the supported file system.

StoRM and cluster file systems

StoRM takes advantage from aggregation functionalities provided by dedicated systems, such as parallel and cluster file systems. A cluster file system allows large numbers of disks attached to multiple storage servers to be configured as a single file system, providing:

  • Transparent parallel access to storage devices while maintaining standard UNIX file system semantics.
  • High-speed file access to applications executing on multiple nodes of a cluster.
  • High availability and fault tolerance .

StoRM and GPFS

The picture below shows a grid site with the storage resources realized with GPFS parallel file system and StoRM as the storage resource manager.

Our solution (Site B) provides a new and efficient way to access the storage in a grid site that leverages on parallel file systems advantages. With StoRM + GPFS solution we are able to provide an efficient, secure and high performing way to access data through a standard POSIX call that job can perform from the Worker Node direct to the file into the Storage Element.

The GPFS cluster is realized using high performing dedicated hardware, and each worker node, in which the computational process take place, belong to the cluster too.

The standard way (Site A) to read and write data during the computational process in a grid site has many drawbacks.

  • For write operation, usually a job produces data in the local worker node disks, and then it have to copy the data into the storage element. During this process, many bad things could happen that prevent the operation to complete sucessfully: not enough free space on disks, disks filled up by different jobs running on the same worker node, network failure during transfer operation, data removed by someone else, etc.
  • For read operation, the only way is to use some external server (I/O Server) that provide a POSIX-like way to access the file using stream from the Storage Element to the requester in the Worker Node.

In the picture:

  • StoRM dynamically manages files and space in the storage system.
  • Applications can directly access the Storage Element (SE) during the computational process.

:documentation:storm-context-cluster.png

Security Framework

Security is a driving feature in StoRM design. Security is based on:

  • VOMS Certificate (X.509 Attribute Certificate Extension).
  • Plug-in to external Authorization sources.
  • File system ACLs to enforce permissions on data.

User authentication is based on the Virtual Organization Membership Service (VOMS), the principals must have a valid proxy to submit the SRM request to the StoRM server. StoRM server is able to interact with different external service for requests authorization.
This external Authorization Sources is used to verify if the user can perform the specified operation on the requested resources. This kind of information can be collected by dedicated service (as the INFN project GPBox, a policy management framework for Grid environments) or retrieved in a more general pourpose catalog (as LFC, the LCG File Catalog, through the ECAR interface). Once the user request has been authorized, StoRM enforce the permission on the data getting the local mapping for the grid user identity and setting a file system ACL for the corresponding user on the specified resource.

StoRM requires ACL capable file systems and it is able to manage the different security approach coming from HEP and Economic and Finance Grid requirements.