DEPRECATEDThis TB has been superseded by TB#530

New Dual-Copy Commands in FTP, HSI, HTAR, and NFT

The High Performance Storage System (HPSS), the archival storage system for the Livermore Computing (LC) open and secure computing facilities, provides different classes of service (COS) for storing files. The Data Storage Group has configured HPSS to use different COS to make the best possible use of available hardware resources. To date, the user interface automatically selects the COS in which to store a file.

Today, files smaller than 256 MB are stored in a COS that provides two copies on separate tapes by default. Small files (0–4 MB) are kept disk resident in the archive as long as possible (on the order of years) to minimize the need to perform costly tape mounts for reads.

HTAR files are by default stored in a COS which is not dual-copied.

An optional dual-copy COS is configured for files larger than 256 MB. This COS is intended only for large, mission critical files. Users can select this COS through new dual-copy commands in the FTP, HSI, HTAR, and NFT user interfaces. (Please do not abuse this capability.)

Dual-Copy Class of Service for Mission Critical Files

Mission critical files are first written to a four-wide striped disk volume. These files are then dual copied to two different types of tape media for added redundancy. LC monitors utilization of the mission critical COS and requests that this service be used only for truly irreplaceable data. Users should note that small files stored as mission critical files—as opposed to the default small COS—will not be kept disk resident as described above.

Examples

FTP

To select the dual-copy COS to store file1 using the FTP user interface:

ftp storage
ftp> site dualcopy
ftp> put file1

To then store file2 to the HPSS default COS (which may or may not be dual-copied based upon file size):

ftp> site nodualcopy
ftp> put file2
ftp> quit 

NFT

To select the dual-copy COS to store file1 using the NFT user interface:

nft
? dualcopy
? put file1

To then store file2 to the HPSS default COS (which may or may not be dual-copied based upon file size):

? nodualcopy
? put file2
? quit

HTAR

To select the dual-copy COS to write file1 and file2 to a new archive called files.tar in the current HPSS home directory using the HTAR user interface:

htar -Y dualcopy -cf files.tar file1 file2

To instead create an archive in the default HTAR COS (which is not dual-copied):

htar -cf files.tar file1 file2

HSI

To select the dual-copy COS to store file1 using the HSI user interface:

hsi
? set cos=dualcopy
? put file1 

To then store file2 to the HPSS default COS (which may or may not be dual-copied based upon file size):

? set cos=nodualcopy
? put file2

To change the COS of existing file3 in HPSS to the dual-copy COS:

? chcos dualcopy file3

To change the COS of existing file4 in HPSS to the HPSS default COS:

? chcos nodualcopy file4
? quit

More Information

A brief summary of FTP, HSI, HTAR, and NFT is available in the man pages. For detailed information, please refer to the reference manuals accessible online at:

https://hpc.llnl.gov/documentation/user-guides/using-ftp-file-transfer-protocol

https://hpc.llnl.gov/documentation/user-guides/using-htar-hpss-tape-archiver

https://hpc.llnl.gov/documentation/user-guides/using-lc-file-systems/using-nft

PDF of TB435 for download and distribution.