EZSTORAGE: Archive and HPSS
This manual introduces tools for effectively storing and archiving files from Livermore Computing (LC) computers by using the High Performance Storage System (HPSS), also referred to herein as "storage." Individual reference manuals provide detailed technical instructions on the tools and techniques introduced in EZSTORAGE: FTP, NFT, HPSS, HTAR, HSI, and Hopper. Additionally, the EZFILES document is a basic guide to using local directories and general file-handling software at LC. For help, contact the LC Hotline via e-mail or via telephone at 925-422-4531.
Reliable, massive, archival data storage is a crucial part of any effective high-performance computing environment. Although the actual disk and tape resources for storing files at LC are large and elaborate, the user interface is constrained to use either the FTP daemon—the protocol for FTP clients or local alternatives to FTP clients, such as NFT and parallel FTP (PFTP), or the Hierarchical Storage Interface Gateway Daemon (hsigwd)—the protocol for HSI and HTAR. Hopper can use either protocol depending on user settings.
Moving files to and from LC production machines, open or secure, is a mainstream storage mission, easy to perform and very reliable. Using file storage in this context avoids quotas on user home directories, avoids purges of files on temporary work disks, and provides virtually unlimited capacity for managing data or computational output. Transfer rates are fast, and FTP connections are very reliable. Customized FTP-daemon interfaces to handle special storage needs (such as NFT for persistent storage transfers or HTAR for very efficiently making large archives directly in storage) are available, too.
Moving files to and from other LLNL machines is more complex. Features of special FTP clients, together with the need to protect unusual file formats during transfer to or from storage, may call for taking extra steps.
Finally, moving files to and from non-llnl.gov machines, such as computers at other sites or the workstations of distant ASC collaborators, is the most complicated of the three situations. It requires either using a two-stage process or running extra enabling software such as VPN (Virtual Private Network). This may involve running FTP twice, or using nonFTP transfers to an LC production machine before actually storing the files with FTP (run on an LC machine).