HTAR Examples

Creating an HTAR archive file

Goal

To create an HTAR archive file in a subdirectory of your storage home directory and use a filter to install several files within that stored archive.

Strategy

1. One HTAR execute line can perform all of the desired tasks quickly and in parallel:

  • The -cvf options create (c) an archive, verbosely (v) report the incoming files, and (f) name the envelope file.
  • The relative pathname case3/myproject.tar locates the archive (myproject.tar) in preexisting subdirectory case3 of your storage home directory (omitting case3/ leaves the archive at the top level of your storage home directory). HTAR will not create case3 by default, however; you must either have previously used FTP's or HSI's mkdir command or else you must invoke -P to explicitly request creation of all needed subdirectories along the archive pathname.
  • File filter tim* selects all and only the files whose names begin with TIM (in the directory where you run HTAR) to be stored in the archive.

2. HTAR opens a preauthenticated connection to your storage (HPSS) home directory and reports its housekeeping activities (very quickly, in lines that overwrite, so you may not notice all of these status reports on your screen).

3. HTAR creates your requested archive and uses parallel connections (but not the PFTP client) to move your requested files directly into it. Directories are handled recursively and directory names (if any) appear with a slash (/) appended to identify them.

4. The last incoming file that HTAR reports is always the 256-byte consistency file by which HTAR coordinates your archive with its external index file.

5. HTAR summarizes the work done (time, rate, amount, thread count), then copies into storage the index file that it made, destroys the local version, and ends.

htar -cvf case3/myproject.tar tim* ---(1)

HTAR: Opening HPSS server connection ---(2)
HTAR: Getting HPSS site info
HTAR: Writing temp index file to /usr/tmp/aaamva09A
HTAR: creating HPSS Archive file case3/myproject.tar ---(3)
HTAR: a tim1.txt
HTAR: a tim2.txt
HTAR: a tim2a.txt
HTAR: a tim3.a
HTAR: a time.txt
HTAR: a time2.gif

HTAR: a /tmp/HTAR_CF_CHK_13805_997722535 ---(4)

HTAR: Create complete for case3/myproject.tar. 399,360 ---(5)
bytes written for 6 member files, max threads: 8
Transfer time: 0.555 seconds (7.257 MB/s)

HTAR: Copying Index File to HPSS...Creating file

Retrieving Files from within an Archive

To retrieve several files from within an existing stored HTAR archive file (without retrieving the whole archive first).

HTAR does not process metacharacters (file filters such as *) itself, but leaves them for the shell to expand and compare with file names in your local directory. Hence, you CANNOT use * to select a subset of already archived files to retrieve. For example, "natural" execute lines

htar -xvf case3/myproject.tar time* [WRONG]
htar -xvf case3/myproject.tar 'time*' [WRONG]

both FAIL to select (and hence to retrieve) any stored files from the MYPROJECTS.TAR stored archive (each yields its own set of error messages). These lines work only accidentally, if you happen to have files with the same name in both your local directory and your stored archive (unlikely except when you are just testing HTAR).

Work-Arounds

A. Type the name of each file that you want to retrieve (at the end of the HTAR execute line).

B. If you have a long list of files to retrieve, or if you plan to reuse the same retrieval list often, put the list of sought files into a file and use HTAR's -L option to invoke that list. You can use HTAR's -t (reporting) option to help generate that retrieval list by reporting all the files you have archived and then editing that report to include only the relevant file names to retrieve. For instance,

htar -tf case3/myproject.tar > hout
grep 'time' hout | cut -c 50-80 > tlist

captures the list of all your stored files in the local file HOUT and then selects just the file names that contain the string TIME for use with HTAR's -L option (here, in local file TLIST). Note that HTAR automatically appends slash (/) at the end of every directory name that -t reports.

C. Use Hopper to run HTAR as a controllee, then select visually the files that you want to retrieve.

1. Once you have laid the groundwork above, a single HTAR execute line can retrieve your specified files quickly and in parallel from within your stored archive:

  • The -xvf options request retrieval/extraction (x), verbosely (v) report the retrieved files, and (f) name the target archive.
  • The relative pathname case3/myproject.tar locates the archive (myproject.tar) in preexisting subdirectory case3 of your storage home directory.
  • The explicit file list (A) or name-containing file (B) selects all and only the files that you want (here, those whose names begin with TIME, a subset of all files stored in this archive in the previous example).

2. HTAR opens a preauthenticated connection to your storage (HPSS) home directory and reports its housekeeping activities (very quickly, in lines that overwrite, so you may not notice all of these status reports on your screen).

3. HTAR uses its external index to locate in the archive the (two) specific files that you requested and then it transfers them by using parallel connections (but not the PFTP client) to your local machine without retrieving the whole archive file.

4. HTAR summarizes the work done (time, rate, amount) and then ends.

htar -xvf case3/myproject.tar time.txt time2.gif ---(A)
   OR
htar -xvf case3/myproject.tar -L tlist ---(B)

  HTAR: Opening HPSS server connection ---(2)
  HTAR: Reading index file
  HTAR: Opening archive file

  HTAR: Reading archive file ---(3)
  HTAR: x time.txt, 1085 bytes, 4 media blocks
  HTAR: x time2.gif, 3452 bytes, 8 media blocks

  HTAR: Extract complete for case3/myproject.tar, ---(4)
        2 files. total bytes read 116,736 in 0.070 seconds (1.669 MB/s)
  HTAR: HTAR SUCCESSFUL

Rebuilding a Missing Index

To rebuild the missing index file for a stored HTAR archive file and thereby (re)enable blocked access to the files within it (and extract some).

1. You try to retrieve all files (-xvf) from the HTAR archive myproject.tar in the case3 subdirectory of your storage home directory.

2. But HTAR cannot find the external index file (here, called myproject.tar.idx) for this archive, and it returns a somewhat cryptic error message, retrieves no requested files, and ends. (File myproject.tar.idx may have been moved, renamed, or accidentally deleted from storage.)

3. So you execute HTAR again with the special action -X (uppercase, not lowercase, eks) to request rebuilding the external index for the (same) disabled archive.

4. HTAR opens a preauthenticated connection to your storage (HPSS) home directory, locates the archive in subdirectory case3, scans (but does not retrieve) its contents, and thereby creates a new myproject.tar.idx file (temporarily on local disk, then moved to the same storage directory as the archive file that it supports). HTAR ends.

5. Now you again try your original (1) file-retrieval request.

6. HTAR opens a preauthenticated connection to your storage (HPSS) home directory and reports its housekeeping activities (very quickly, in lines that overwrite, so you may not notice all of these status reports on your screen).

7. HTAR uses its (newly rebuilt) external index to locate the files within the archive and transfers them by parallel connections to your local machine (it transfers all of them because there is no filelist on the execute line).

8. HTAR summarizes the work done (time, rate, amount) and then ends.

htar -xvf case3/myproject.tar ---(1)

  HTAR: Opening HPSS server connection
  HTAR: Getting HPSS site info
  ERROR: Received unexpected reply from server: 550 ---(2)
  ERROR: Error -1 getting Index File attributes...
  HTAR: HTAR FAILED
  ###WARNING htar returned non-zero exit status.
             72 = /usr/local/bin/htar.exe...

htar -Xf case3/myproject.tar ---(3)

  HTAR: Opening HPSS server connection ---(4)
  HTAR: Reading archive
  HTAR: Copying Index File to HPSS... creating file
  HTAR: HTAR SUCCESSFUL

htar -xvf case3/myproject.tar ---(5)

  HTAR: Opening HPSS server connection ---(6)
  HTAR: Reading index file
  HTAR: Opening archive file

  HTAR: Reading archive file ---(7)
  HTAR: x tim1.txt, 3503 bytes, 8 media blocks
  HTAR: x tim2.txt, 4310 bytes, 10 media blocks
  HTAR: x tim2a.txt, 5221 bytes, 12 media blocks
  HTAR: x tim3a., 5851 bytes, 13 media blocks
  HTAR: x time.txt, 1085 bytes, 4 media blocks
  HTAR: x time2.gif, 3452 bytes, 8 media blocks

  HTAR: Extract complete. total bytes read: ---(8)
       28,160 in 0.141 seconds (0.200 MB/s)
  HTAR: HTAR SUCCESSFUL