LC Hotline: 2-4531

From offsite: (925) 422-4531

 

Hours

Monday–Friday
8am–12pm, 1–4:45pm
B453 R1103 | Q-clearance area

 

 

Storage Interfaces

FTP

FTP is the standard interface to HPSS. When you run FTP (on an OTP or Kerberos-passworded LC machine) with storage as the target host, access is "preauthenticated" and you are not prompted for your password. Also, on all LC production machines (but not necessarily on other LC machines), a parallel FTP client (equivalent to PFTP) is the default. All files that are 4 MB or larger automatically move to or from storage using parallel FTP. For more information about the FTP file-transfer utility, consult the FTP Usage Guide.

Note: Because the storage FTP daemon (based on the HPSS version) behaves differently from the other LC FTP daemons (based on the WU FTP daemon), users should be aware that "m" commands (mdelete, mget, mput, etc.) may produce unintended results. These "m" commands process multiple files by using as their argument either an explicit file list or a file filter (an implicit file list specified with one or more UNIX wildcard or metacharacters.) The best method for checking the behavior is to type ls pattern where pattern is what will be used with the "m" command. If ls pattern returns something unexpected, the pattern should be reformulated.

NFT

NFT is a locally developed file transfer tool. Although NFT uses standard FTP daemons to carry out its file transfers, it offers enhanced features.

  • A special NFT server preauthenticates all NFT transfers, so all NFT executions are passwordless.
  • NFT elaborately tracks and numbers all transfers. It automatically persists if system problems delay storing any file, and it keeps detailed records of your file-storage successes and problems.
  • Input from and output to files is easy, and NFT's command syntax (unlike FTP's) lends itself to practical use in scripts and batch jobs.
  • Some NFT commands especially facilitate transfers to and from storage (so some users regard NFT as primarily a file-storage rather than a general file-transfer tool). Also, NFT automatically "routes" storage-related file transfers to take advantage of fast, jumbo-frame network connections whenever they are available (especially helpful for transfers between the Lustre parallel file system and storage).

For a complete analysis of NFT syntax and special features, along with a thorough alphabetical command dictionary, consult the NFT section of this manual.

HSI

HSI provides a UNIX shell-style interface to HPSS and supports several of the commonly used FTP commands with the following differences:

  • The dir command is an alias for ls in HSI. The ls command supports an extensive set of options for displaying files, including wildcard pattern-matching and the ability to recursively list a directory tree.
  • The put and get family of commands support recursion.
  • There are "conditional" put and get commands (cput, cget).
  • The syntax for renaming local files when storing files to HPSS or retrieving files from HPSS is different from FTP. With HSI, the syntax is always local_file : hpps_file, and multiple such pairs may be specified on a single command line. With FTP, the local file name is specified first on a put command and second on a get command. For example, when using HSI to store the local file "file1" as HPSS file "hpss_file1" and then retrieve it back to the local file system as "file1.bak", the following commands could be used:

    put file1 : hpss_file1
    get file1.bak : hpss_file1

    With FTP, the following commands could be used:

    put file1 hpss_file1
    get hpss_file1 file1.bak
  • The "m" prefix is not needed for HSI commands; all commands that work with files accept multiple files on the command line. The "m" series of commands are intended to provide a measure of compatibility for FTP users.

See more detailed information about HSI usage, including a command dictionary.

Hopper

Hopper offers a graphical interface to storage and other LC resources, including support for simple drag-and-drop file-transfer services using FTP, NFT, HSI, HTAR, etc. By invoking Hopper, you can do many file-management tasks graphically, including:

  • Transfer files to and from storage.
  • Synchronize directories between LC resources or your desktop and storage.
  • Search for files in storage by name, size, age, or other criteria.
  • Create, view, and extract from HTAR archives.

More general background information on Hopper is available at the Hopper Web site. See "Getting Started" for instructions on how to download Hopper to your local desktop machine.

HTAR

On LC production machines (but not at other ASC sites), HTAR is a separate, locally developed utility program that serves as a special-purpose front end to the parallel FTP daemons for storage access. HTAR combines a flexible file bundling tool (like TAR) with fast parallel access (it acts as an alternative to the PFTP client) to storage that lets you store and selectively retrieve even very large sets of files very efficiently.

HTAR's enhanced features include the following:

  • Imposes a 100 TB limit on the total size of the archives that it builds and accepts input files (archive members) as large as 68 GB.
  • Uses a TAR-like syntax and supports TAR-compatible archive files.
  • Bundles files in memory using multiple concurrent threads and transfers them into an archive file built directly in storage by default (to avoid needing extra local online disk space).
  • Takes advantage of available parallel interfaces to storage to provide fast file transfers.
  • Uses an external index file to easily accommodate thousands of small files in any archive and to support retrieval of specified files from within a still-stored archive without first retrieving the entire archive from HPSS. (Warning: You can use filters such as * to create an HTAR archive, but you cannot reliably use filters to retrieve files from within an already stored HTAR archive. See the "Retrieving Files" section of the HTAR section of this manual for possible workarounds.)
  • Allows easily building and storing incremental archives (consisting of only recently changed files).

Complete details about using HTAR are available in the HTAR section of this reference manual.