LC Hotline: 2-4531

From offsite: (925) 422-4531

 

Hours

Monday–Friday
8am–12pm, 1–4:45pm
B453 R1103 | Q-clearance area

 

 

Parallel FTP Service

Parallel FTP service is available between each LC production machine and (both OCF and SCF) storage, as well as between pairs of LC production machines themselves. In all cases where parallel service is available, it is automatic. Multiple "command completed" messages (one for each parallel stripe) betray the parallel transfer of large files with FTP. (HTAR also automatically uses parallel transfers but does not execute the PFTP client.)

Also, the login nodes on some LC production machines are connected to "jumbo-frame gigabit Ethernet links" for fast network traffic. The OCF machines oslic and rzslic and SCF cslic have jumbo-frame links, and are best used for data transfer to storage. Naturally, the best file-transfer rates occur when you get parallel FTP between a pair of machines that also has the fast jumbo-frame links. For example, NFT automatically routes storage transfers to a cluster's login nodes to take advantage of the jumbo-frame links.

LC has installed jumbo-frame gigabit Ethernet links on both OCF and SCF storage systems and the login nodes of most LC clusters (AIX and Linux/CHAOS). Use of these jumbo-frame network links for faster file transfers is automatic among those LC computers that have them installed. FTP, HSI, HTAR, and NFT all utilize the jumbo-frame gigabit Ethernet links automatically.

Parallel Transfers

The default FTP client on all LC production machines (but not necessarily on special-purpose machines) is a locally developed version that enables you to transfer data in parallel. When parallel FTP transfers are possible, they occur automatically. The FTP server ("daemon") on your destination (target) machine determines whether or not the file transfer is automatically parallel. This is the case for all LC production clusters (AIX or Linux/CHAOS) and storage. For all files over 4 Mbytes, FTP file transfers to storage from all LC production machines (both directions) are automatically parallel (OCF and SCF). Transfers originating on those machines with jumbo-frame gigabit Ethernet links also automatically use those links for even faster data movement. FTP's parallel command now simply reports the current parallel stripe width and block size.

PFTP Commands

The PFTP client offers a number of extra commands (beyond the usual set offered by FTP) to specifically manage parallel file transfers (for example, pget and mpget perform parallel gets). On LC production machines these special PFTP commands are quite unnecessary because parallel transfers occur automatically where they are possible. At other (ASC tri-lab) sites, you may need to remember the special PFTP commands to perform parallel file transfers (especially to storage). See LC's HPSS User Guide for information on extra PFTP commands.

Optional Extras

LC's parallel FTP client is more verbose than the standard FTP client during file transfers. Parallel FTP users may want a complete record of each verbose FTP dialog in their batch log files. The child mode execute-line option (-c) causes all interactive output to be sent during batch runs of FTP, and the echo mode (-e) option copies FTP input commands into your batch output. Thus, running FTP with the execute line

ftp -ce remotehost

will preserve all the details of a parallel session even within a batch job. For more information on these and other FTP-related options, consult the FTP main pages.

FTP sessions with storage (and with some other target machines) are fully preauthenticated and do not prompt for your user name, while in other cases FTP returns a

Name (host:username):

prompt to which you must reply to continue. Parallel users who want to eliminate this Name: prompt from all sessions (including batch sessions) can install a file called .netrc in their (global) home directory, containing the following lines:

    default
    login username
    macdef init
    binary
    <empty line>

where the last line in the .netrc file is present but blank. This will put you into binary mode every time, which is fine as there is no need to use the default ASCII mode