NOTEThis change will not take effect until 4/30 or 5/1; until then, use TB#435.
New Dual-Copy Commands in FTP, HSI, HTAR, and NFT
The High Performance Storage System (HPSS), the archival storage system for the Livermore Computing (LC) open and secure computing facilities, provides different classes of service (COS) for storing files. The Data Storage Group has configured HPSS to use different COS to make the best possible use of available hardware resources. To date, the user interface automatically selects the COS in which to store a file.
Today, files smaller than 256 MB are stored in a COS that provides two copies on separate tapes by default. Small files (0–4 MB) are kept disk resident in the archive as long as possible (on the order of years) to minimize the need to perform costly tape mounts for reads.
An optional dual-copy COS is configured for files larger than 256 MB. This COS is intended only for large, mission critical files. Users can select this COS through new dual-copy commands in the FTP, HSI, HTAR, and NFT user interfaces. (Please do not abuse this capability.)
Dual-Copy Class of Service for Mission Critical Files
Mission critical files are first written to a four-wide striped disk volume. These files are then copied to tape media utilizing RAIT (Redundant Arrays of Independent Tape) which is an HPSS software technology for striping data on tape with the added benefit of parity for redundancy. LC monitors utilization of the mission critical COS and requests that this service be used only for truly irreplaceable data. Users should note that small files stored as mission critical files—as opposed to the default small COS—will not be kept disk resident as described above.
Examples
FTP
To select the dual-copy COS to store file1 using the FTP user interface:
ftp storage ftp> site dualcopy ftp> put file1
To then store file2 to the HPSS default COS (which may or may not be dual-copied based upon file size):
ftp> site nodualcopy ftp> put file2 ftp> quit
NFT
To select the dual-copy COS to store file1 using the NFT user interface:
nft ? dualcopy ? put file1
To then store file2 to the HPSS default COS (which may or may not be dual-copied based upon file size):
? nodualcopy ? put file2 ? quit
HTAR
To select the dual-copy COS to write file1 and file2 to a new archive called files.tar in the current HPSS home directory using the HTAR user interface:
htar -Y dualcopy -cf files.tar file1 file2
To instead create an archive in the HPSS default COS (which may or may not be dual-copied based upon file size):
htar -cf files.tar file1 file2
HSI
To select the dual-copy COS to store file1 using the HSI user interface:
hsi ? set cos=dualcopy ? put file1
To then store file2 to the HPSS default COS (which may or may not be dual-copied based upon file size):
? set cos=nodualcopy ? put file2
To change the COS of existing file3 in HPSS to the dual-copy COS:
? chcos dualcopy file3
To change the COS of existing file4 in HPSS to the HPSS default COS:
? chcos nodualcopy file4 ? quit
More Information
javascript:void(0);
A brief summary of FTP, HSI, HTAR, and NFT is available in the man pages. For detailed information, please refer to the reference manuals accessible online at:
https://hpc.llnl.gov/documentation/user-guides/using-ftp-file-transfer-protocol
https://hpc.llnl.gov/documentation/user-guides/using-htar-hpss-tape-archiver
https://hpc.llnl.gov/documentation/user-guides/using-lc-file-systems/using-nft
PDF of TB435 for download and distribution.