Data Management Tools & Projects

Here are a number of the more popular tools you'll find on LC systems for moving data between different destinations.

ToolDescriptionWhen to UseHow to Learn More
dcpCLI tools developed at LLNL for moving data between two Lustre filesystems on the same host. Uses the MPI protocol and is part of a suite of tools called mpiFileUtils.When you need to move a large amount of data from one Lustre file system to another.
ftpStandard Linux/Unix CLI tool for moving data between systems and between LC systems and HPSS. Neither the data nor authentication are encrypted. The ftp protocol is one of the older internet protocols and is sometimes used for transferring non-sensitive data between sites. The version on LC systems does parallel transfers to or from HPSS and between LC systems.When moving a small number of  large files (> 256 MB) into or out of HPSS, or between systems.
hopperA GUI tool developed at LLNL and available on all LC systems that provides an "umbrella" interface to all of the tools listed here, plus more.Whenever you prefer to use a GUI for managing your data.
hsiCLI tool provided with HPSS, the Hierarchical Storage Interface (HSI) communicates with HPSS via a UNIX shell-style interface that makes it easy to transfer files and manipulate files and directories using familiar UNIX-style commands. HSI supports recursion for most commands, as well as csh-style support for wildcard patterns and interactive command line and history mechanisms.When you need advanced features or you want to use one of the standard HPSS tools for interacting with your archived data.
htarCLI tool provided with HPSS for moving data between LC clusters and HPSS. Similar to GNU 'tar' in how it works, except the archive file lives in HPSS instead of on local disk. This is the fastest way to get data into HPSS, especially when working with small or medium sized files.When copying a large number of small to medium files from an LC system into HPSS, especially if you want to later retrieve those files efficiently.
nftScriptable CLI tool developed at LLNL for moving data between LC systems and HPSS. Provides an interface to the Endeavor server, which offers persistent transfers that will continue to try even if hosts, file systems, or the storage system goes down.When you want to "set and forget" a transfer into or out of HPSS.
rsyncStandard Linux/Unix CLI tool that copies files either to or from a remote host, or locally on the current host. Most useful for synchronizing two directories by comparing size, attributes, and or contents of the source and sink files before deciding if it needs to be copied.When synchronizing directory trees between different hosts or different filesystems.
  • man rsync
sftp / scpStandard Linux/Unix CLI tools for securely moving data between systems. Based on the SSH protocol, so data and authentications are encrypted.When moving data between a desktop and LC system, or between LC systems, and you want it to be encrypted in flight.
  • man sftp
  • man scp