Technical Bulletin #436: HSI—Hierarchical Storage Interface
The Livermore Computing (LC) Data Storage Group (DSG) recently released the utility program called HSI (Hierarchical Storage Interface) on all open (OCF) and secure (SCF) production machines. HSI is a powerful interface for users of the High Performance Storage System (HPSS). It is intended to provide a familiar UNIX-style environment for working within the HPSS environment, while automatically taking advantage of the power of HPSS (e.g., for high speed parallel file transfers) without requiring any special user interaction, where possible.
How to Use HSI
On any LC production system machine simply run:
Upon startup, HSI will first indicate that it is authenticating your identity, then a line with your user name, uid, an HSI version number, and build date is returned, followed by a question mark prompt.
HSI has many features, including some that are unique among LC HPSS user interfaces:
- Familiar UNIX-style command interface, with commands such as cd, chgrp, chmod, chown, du, find, ls, etc.
- Recursive option is available for most commands; including the ability to copy an entire directory tree to or from HPSS with a single, simple command.
- Interactive “history” and command-line editing.
- Interactive, batch, or “one-liner” execution modes.
- Ability to distinguish whether files are resident on HPSS disk cache or on tape (i.e., purged from disk).
- Ability to automatically retrieve files in tape-order, improving performance for stage/get operations.
- Ability to change the Class of Service of an HPSS file (typically useful when mission critical data require dual-copy). Ability to interactively pipe data into or out of HPSS, using filters such as “TAR.”
- Conditional put and get operations, including ability to update based on file time stamps.
- Automatically uses HPSS parallel I/O features for file transfer operations.
- Command aliases and abbreviations.
To put a file with the same name locally as in storage:
? put filename
To put a file in storage with a different name than the local name:
? put localname : storagename
To get a file from storage with a different name than the local nameL
? get storagename : localname
To list a file and determine if it is on disk or tape:
? ls -U filename -rw-r----- 1 user group 140 6859 TAPE 1073741824 Aug 16 2005 filename
More information is available with the -X option instead of -U. The two lines for storage levels 0 and 1 are most informative. It is possible to observe a large file being staged from tape and to monitor its progress.
? ls -lX filename -rw-r----- 1 user group 140 6950 TAPE 1073741824 Aug 16 2005 filename Storage VV Stripe Level Count Width Bytes at Level ---------------------------------------------------------------------------- 0 (disk) 0 1 (no data at this level) 1 (tape) 1 1 1073741824 . . .
To display the number of bytes contained in all files, and recursively, directories within a specified directory or file name:
? du –k 144414335 /users/u39/username/ ----------------------- 144414335 total 1024-byte blocks, 501 Files (147,880,278,678 bytes)
To conditionally save a file to HPSS only if the local file time stamp is newer than the HPSS file:
hsi put –u filename
To specify a very large number of input files yet avoid an HSI command line too long for the shell to accept:
hsi < input.file
For details on HSI’s capabilities and usage, see the HSI man page or visit the HSI Web site at http://www.mgleicher.us/index.html/hsi
A brief syntax summary is available in the HSI man pages or by using the command
PDF of TB436 for download distribution.