How to Run NFT

Basic Execution

To run NFT on any LC machine where it resides, type

nft [options]

where NFT's possible execute-line options are

-i

ignores any .nftrc file. By default, as soon as it starts to execute, your NFT client reads the file .nftrc (if any) in your home directory and immediately processes any NFT commands it finds there, echoing their normal responses (if any) to your terminal. This run-control file typically contains requests for nondefault environment-variable settings or for automatic logging of your NFT messages.

-n

outputs a newline character after each NFT prompt (omitted by default). This is intended to facilitate the handling of PERL scripts, but it is not recommended for ordinary interactive use.

NFT prompts for more input with the string nft> and you terminate your NFT client by typing QUIT (or, less elegantly, CTRL-C). Even after you have stopped running your NFT client, however, the NFT server will persistently execute any jobs (file-transfer requests) that have not yet completed (you can only kill your incomplete NFT jobs by using NFT's ABT command).


Basic NFT Features

DAEMONS.

Although NFT uses its own special server to schedule file-transfer requests and to persistently track them, it uses the standard FTP daemons on the sending and receiving machines to actually carry out your file transfers. Hence, some NFT commands (such as CHGRP or LN) may fail on some machines because the local FTP daemons do not support them there.

BINARY TRANSFER.

All NFT file transfers use the FTP binary (image) mode. You cannot change to ASCII mode with any NFT command.

PASSWORDS.

The NFT server preauthenticates your access to the machines where NFT works. Hence, all NFT file transfers are passwordless.

LIMITS.

The longest path name that NFT accepts is 1023 characters. (Remember also that many UNIX utilities limit names to 16 characters.) The largest file that you can store using NFT is 10 Tbyte (use HTAR to store and manage very large numbers of related files).

SYNTAX.

To specify the donor and target locations of files to transfer, NFT primarily uses a prefix or sentinel notation, somewhat like that used by SCP, rather than the login-based approach that FTP requires. For details on and examples of this prefix file-specification syntax, see the NFT Path Name Syntax section. OPEN command for a way to make NFT somewhat mimic the FTP login approach.

SPECIAL CHARACTERS.

Several characters have special roles when used with NFT:

semicolon (;)

NFT recognizes the semicolon (;) as a command separator on its execute line or in response to any NFT prompt (e.g., CLOBBER;PWD).

filters ( ? * [a-b] )

The standard UNIX file filter (wild card) characters (? for single characters, * for any string, and [a-b] for end points of a specified range) are all accepted within file names by most NFT commands and can be used literally only if quoted (exceptions, where the result would be ambiguous, are noted in the description of specific NFT commands below).

filelists ( {a,b,c} )

NFT accepts these as a list of itemized file names a, b, and c.

other nonalphanumerics

- | ~

cannot appear as the first character in any file name, but can be used in other positions.

, : }

must be quoted if they appear in any file name (e.g., 'a:b'), because each otherwise has a special meaning for NFT.

quotes ( ' " )

Quotes have two special roles for NFT:

Quoted commands

Matched quotes surrounding commands only on NFT's execute line allow promptless execution (e.g., see the Input Files section). NFT rejects quotes around any commands after its prompt as a syntax error.

Quoted file names

Matched quotes surrounding any file name in an NFT command (e.g., PUT 'a:b') can protect imbedded special characters in the name, allowing them to behave as ordinary alphanumeric characters. But note that this quote protection has two important limits:

  1. Quotes do not protect - | ~ in the first-character position.
  2. Quotes protect file-filter characters from NFT but not from subsequent special handling by many FTP daemons, who treat them as filters anyway.

COMMANDS.

To request and monitor file transfers, NFT uses interactive commands and environment-variable settings quite like FTP. However, some commands (e.g., GET) have specialized, storage-only roles for NFT that differ from their FTP roles. For details on the NFT commands, see the Command Summary or the much longer Command Dictionary. When NFT's interactive commands have multiple, nonexclusive suboptions, you must concatenate all your chosen suboptions with a single hypen (-) sentinel, not flag each with its own sentinel as UNIX usually allows. Thus, for example,

dir -FPt

FILES.

To handle special situations, NFT can accept its input from files and send its output to files. For usage instructions, see the NFT Input and Output Filessection.

TRANSFER RATES.

To take advantage of the significantly faster file transfers (to or from storage only) that jumbo-frame network connections enable, NFT automatically routes compute-node transfers to/from storage through the login nodes on many LC clusters. For more details, for the implications for LC's parallel file systems, and for ways to check or disable this routing, see NFT's ROUTING command.

JOB TRACKING.

NFT goes far beyond SCP or FTP in the elaborateness of its job tracking. Relevant features include uniquely numbering each job; commands to report job status before, during, and even after completion; user control of NFT's interactive messages about job progress; and ways to create and monitor "sessions" of related jobs. For details, consult the Job Status and Reporting section.

CLASSES OF SERVICE (STORED COPIES).

See the SETCOS section of the HPSS Manual for a detailed discussion of HPSS "class of service" policy. STATUS reports the current COS requested, and you can report an already stored file's class of service by using the -h suboption of NFT's DIR command. NFT handles new and previously stored files differently, however, in regard to COS values.

ERROR MESSAGES.

Most NFT messages about user errors are sequentially numbered along with other NFT responses and begin with the string "error":

n.0 error explanation

If you transfer many files at once, NFT explicitly distinguishes between "no clobber failures" (overwrites that you prevented) and other failures when it reports errors. Note that, if HPSS storage is down for planned maintenance, then attempts to store files using NFT yield a different error message with this special format:

[SCF|OCF] HPSS Storage is down for maintenance.
*** Try again later.

Scripts that execute NFT should check for this special "three-star" error message and avoid needlessly resubmitting NFT jobs (storage requests) once it has been detected.

GRAPHICAL INTERFACE.

If you prefer a graphical interface to (mouse-oriented controller for) NFT, execute HOPPER on any LC production machine and select NFT from HOPPER's CONNECT menu.


Basic Examples

This annotated example shows typical file transfers using NFT.

GOAL:

To transfer several files among (open) LC machines using NFT without logging on to all of the machines.

STRATEGY:

  1. Start NFT.
  2. For convenience, change the working directory on YANA to /p/lscratchb/jfk (you could use path names later and skip this step).
  3. Transfer (outward copy) local file t1 to /p/lscratchb/jfk/t2 on YANA.
  4. Transfer (inward copy) file /p/lscratchb/jfk/t3 from YANA to local file t4.
  5. Without logging on to either YANA or ATLAS, transfer (copy) /p/lscratchb/jfk/t3 from YANA to /usr/tmp/t6 on ATLAS.
  6. Use storage-default command PUT to transfer file t1 from the client machine (where NFT runs) to storage.llnl.gov as file t2. Note that NO hosts are specified in this command because everything is defaulted.
  7. Try to retrieve file t8 from storage to local file t4 using the storage-default GET command. Because NFT's default environment is NOCLOBBER, this attempt fails because t4 already exists as a result of step (4) above. You could use the CLOBBER option next, to allow this overwrite, or...
  8. Use GET to retrieve t8 from storage with no name change (and hence no overwriting of t4).
nft ---(1)

nft>cd yana:/p/lscratchb/jfk ---(2)
remote host yana: wd is /p/lscratchb/jfk

nft>cp :t1 yana:t2 ---(3)
1.0. 95 bytes received in 0.1 seconds
(0.7 Kbytes/s) from /g/g0/jfk/t1 to /p/lscratchb/jfk/t2
1 entry copied /g/g0/jfk/t1

nft>cp yana:t3 :t4 ---(4)
2.0. 98 bytes received in 0.1 seconds
(1.1 Kbytes.s) from /p/lscratchb/jfk/t3 to /g/g0/jfk/t4
1 entry copied /p/lscratchb/jfk/t3

nft>cp yana:t3 atlas:/usr/tmp/t6 ---(5)
3.0. 98 bytes received in 0.4 seconds
(0.2 Kbytes/s) from /p/lscratchb/jfk/t3 to /usr/tmp/t6
1 entry copied /p/lscratchb/jfk/t3

nft>put t1 t2 ---(6)
4.0. 95 bytes sent in 1.0 seconds
(0.1 Kbytes.s) from /g/g0/jfk/t1 to ~/t2
1 entry copied /g/g0/jfk/t1

nft>get t8 t4 ---(7)
5.0. error. Cannot clobber existing
sink /g/g0/jfk/t4

nft>get t8 ---(8)
6.0. 95 bytes received in 1.8 seconds
(0.1 Kbytes/s) from ~/t8 to /g/g0/jfk/t8
1 entry copied ~/t8
nft>quit