LC Hotline: 2-4531

From offsite: (925) 422-4531

 

Hours

Monday–Friday
8am–12pm, 1–4:45pm
B453 R1103 | Q-clearance area

 

 

HTAR

HTAR ("HPSS Tape Archiver") is an LC-designed TAR-like utility program that makes TAR-compatible archive (library) files but with High Performance Storage System (HPSS) support and enhanced archive-management features. HTAR's enhancements include its ability to:

  •  Bundle many small files together in memory (without using more local disk space, as standard TAR requires) for more efficient handling and transfer.
  •  Send the resulting large archive file directly to storage without your needing to invoke FTP separately.
  •  Retrieve individual files from a stored archive without moving the whole large archive back to your local machine first (or, optionally, without even staging the whole archive to disk).
  •  Accelerate transfers to and from storage by deploying multiple threads and by automatically using as many parallel interfaces to storage as are available on the (production) machine where it runs.
  •  Easily create incremental backup archives to supplement a master archive with (only) files recently changed (with -n).

The TAR and HTAR Compared section compares traditional UNIX TAR with LC's enhanced HTAR feature by feature to reveal the value of this added tool, but in general HTAR maintains full output compatibility with the POSIX 1003.1 standard TAR format while successfully archiving hundreds or even thousands of incoming files and handling files of greatly mixed sizes or types. In most cases, creating a stored archive directly using HTAR will be much faster than either creating a local TAR file and then copying it to storage with HSI or piping TAR output into an FTP connection to HPSS or into HSI.

HTAR can store archive-member files as large as 68 GB. There is no maximum size for a whole HTAR archive other than site-imposed restrictions (about 10.7 TB) or amount of space available. HTAR makes two copies of each stored archive by default only for files up to 256 MB; you can request dual-copy storage (for extra safety) of a mission critical archive of any size by using HTAR's -Y dualcopy option. (The HTAR -Y dualcopy option can also be specified to pick the class of service (COS) for either the archive file, the index file, or both, or to specify automatic COS selection.) NFT's DIR -h command-with-option combination reveals the COS value of stored files (in output column 3). FTP and HSI can reveal COS, too.

Because HTAR combines two features usually separate (file bundling and file storage), the How HTAR Works section explains the relationship among the three files (the archive file, the index file, and the consistency file) that HTAR uses. See How to Run HTAR for information about how to run HTAR, common error conditions, known limitations (with work-arounds), and HTAR environment variables. The HTAR Options section describes the function of each HTAR option (distinguishing the required action options from the control options). HTAR Examples gives annotated step-by-step examples of how to use HTAR to handle common file-archiving tasks and problems.

HTAR users may also benefit from familiarity with another LC-developed specialty tool that provides nonstandard file-handling and file-transfer features linked to file storage, namely NFT (see the NFT Reference Manual for details). HTAR itself does not, however, use NFT's "persistence" mechanisms to manage file-storage delays. For a general introduction to LC storage tools and techniques, see EZSTORAGE. Hopper, LC's graphical file-handling interface, can also serve as a front end for HTAR in situations where Hopper's scalability limits are not too severe. Also of interest to the HTAR user is the HSI utility, which provides a user-friendly UNIX-style interface to storage. HSI can recursively store, retrieve, and list entire trees with a single command. Consult the HSI manual for details.

For help, contact the LC Hotline at 925-422-4531 (OCF e-mail: lc-hotline@llnl.gov, SCF e-mail: lc-hotline@pop.llnl.gov).