Hopper is a powerful interactive tool that allows users to graphically move, copy, find, delete, and otherwise operate on files. Users can connect to and manipulate resources using ftp, ssh, sftp, htar, Endeavor (nft), and other protocols.
Operations include changing permissions, checking access, searching using wildcards, synchronizing directories, and many others.
- Drag-and-drop interface for file transfers between local and remote hosts or between remote hosts
- Diverse protocol support, including ftp, ssh, sftp, HPSS client API, htar, hsi, jar, zip, tar, and Endeavor (NFT)
- Simultaneous transfers, searches, and other operations
- Search by content or by filename using wildcards, regular expressions, and other criteria
- Synchronize directories between any two systems
- Perform high level data management tasks, including access checking, permission changing, etc.
- Advanced directory browsing - Hopper allows users to view directories in three different formats—tabular, content, and detailed, with a wide variety of sorting and display options. A number of sophisticated mechanisms are provided for traversing directory structures. Also, the user has the option of caching directory lists from remote hosts during a session, allowing fast redisplay of previously visited directories.
- Drag-and-drop file transfer - Using drag-and-drop, Hopper provides a powerful file transfer capability. It is just as easy to copy a file between two remote systems as it is to copy a file from the local host to itself. Files and directories may be transferred either singly or in groups with just a few movements of the mouse.
- Authentication management - Hopper does a lot of work to reduce the password prompts you see. Freed connections are cached and re-used, and authentication information is stored during the session and re-used whenever possible.
- Directory synchronization - Hopper can synchronize two local or remote directories, and supplies a wide variety of options for customizing the operation. By default, old files and directories in the destination folder are replaced by newer ones in the source folder. This behavior can be altered, for instance, by specifying:
- File patterns to exclude from the operation (e.g., "*.bak" or "core")
- That you only want to transfer files and directories containing a certain pattern (e.g., "*.java")
- That you want an exact mirror to be created, even if it means removing files
- Powerful search - One of Hopper's most powerful features is its ability to search directory structures for files or directories matching specified criteria. The user can search by name using literal patterns, wildcards, or regular expressions, or can search by other file attributes such as file size, creation date, or owner. Users can now search on the contents of files on Unix hosts, and can combine filename and content searches (e.g., find all .c files that contain the string "function1"). The efficient Unix search commands—find, locate, and grep—are used when possible because they produce rapid results, but searches can be done just as easily on any non-Unix host to which Hopper can connect. A search can be run concurrently with transfers or other searches.
- File viewing - Viewing local or remote files is as easy as double-clicking on the file. For remote files, this will transfer the file to a temporary directory on the local host for viewing. For local files, viewing is done in place. Several types of files can be viewed using Hopper's built-in viewer; this includes jpeg, gif, and png image files, plus text and raw binary files. The viewer is chosen based on the file suffix (e.g, ".jpg"). Alternative viewers can be assigned via the preferences. Files can also be viewed by performing a "right click" on the file of interest and choosing "View With" from the pop-up menu.
- File comparison - Hopper users running on Unix systems can now easily initiate file comparisons by simply dragging one file icon onto another. This will copy remote files (if necessary) to the local host and either launch an external differencing tool, such as xdiff, or use its own built-in comparison capability.
- Favorites - The favorites feature allows Hopper users to quickly connect to favorite hosts and directories. With the associated autoconnect option, users can do things such as automatically open a window to a certain directory in storage when Hopper starts up.
- Data aggregation - Hopper can perform data aggregation during a transfer: users can create a tar or indexed tar file on a remote host from local data. This aggregating style of transfer is up to 10 times faster than copying files individually with ftp.
- Xterm launching - When running Hopper on an X Window system and connecting to a remote host via SSH, you can launch xterm windows on that remote host with a click of a button. For convenience, xterm will be started in the same directory that the Hopper browser window is viewing.
- History mechanism - Hopper's history mechanism automatically remembers search expressions, previously visited directories, and host and user names used in connecting to remote hosts. This mechanism, along with other features, makes it painless to establish a connection with remote hosts.
- Directory manipulation - A number of directory manipulation features are provided for operating on selected entries. There are several ways to select a group of directory entries to operate on, including direct toggling of entries, sweeping out a rectangular area, and wild-card selection. Once entries are selected, the user can rename entries, delete entries (including entire subdirectories), or move entries between directories within the same host. The user is also able to create directories, both on local and remote hosts.
- Configurable interface - Hopper presents the user with a graphical user interface for easily tailoring the program's behavior. For example, the user can specify whether or not confirmation dialogs appear before certain operations are done. The user can also provide command line options to various programs initiated by Hopper, including ssh and xterm. There are also handy controls for setting the color and font used by Hopper's browser windows, as well as the double-click speed.
Hopper uses abstraction layers to easily extend its capabilities.
The FileSystem interface connects to underlying technologies such as FTP, SSH, TAR, and so on.
The Metadata interface connects to local and remote metadata sources.
The Copy interface connects FileSystems during transfers, providing recursion, two- or three-hop transfers, etc. as needed. The power of this architecture is exhibited when transferring items between dissimilar FileSystems. For instance, one can transfer from a remote tar file on one host to another remote host connected via ftp.
Consult Hopper Help for complete Hopper usage details.
You can also learn more about Hopper by reading the following sections of this user guide:
If you have feedback, comments, or bug reports for Hopper, please send mail to email@example.com.
Q. What is Hopper?
A. Hopper is a powerful, interactive, cross-platform tool for performing file management and data transfer tasks. Through Hopper's graphical interface, a user can perform operations on files and directories that reside on the local host, on remote hosts, and on remote resources such as HPSS.
Q. How do I submit feature requests, bug reports, or other feedback?
Q. Will Hopper run on my Mac/PC/Unix desktop?
A. Probably. Hopper runs on Mac OS X, Windows, and most variants of Unix. Basically, any platform that supports Java JDK 1.7 or later will likely run Hopper.
Q. How do I open multiple windows to one host?
A. There are several ways to do this. To open another window displaying the same directory, just choose the "Display in New Window" option from the File menu. To create a new window showing a sub-directory, single-click the middle mouse button on the icon or name of the directory you want to display in another window. This opens a new window displaying that directory's contents. You can also use the right mouse button over a directory entry and choose "Display in New Window" from the menu. (On systems with a one-button mouse, a right click can be generated by holding the Control or Command key down while clicking the mouse.)
Q. How do I transfer files or directories?
A. Transferring files and directories is done via drag and drop. Use the mouse to select the items you want to transfer, click and hold the left mouse button, drag the cursor to the destination of your transfer in another window, then release the mouse button. The cursor should change to a "happy face" when you're over a valid destination. Once you release the cursor a menu will pop-up showing your options. Choose "Copy Here" to initiate the copy.
Q. How do I move files rather than copy them?
A. Follow the directions from above for copying, but choose "Move Here" from the drop menu. Note that moves are only possible within a given host, i.e., you cannot move files from one host to another. If you need this behavior you can copy the files and then delete the originals.
Q. When in detailed view, how do I copy into a sub-directory?
A. In detailed view, you must drag onto the folder icon of the sub-directory when you want to move or copy into it. Unlike in tabular view, dragging onto the neighboring whitespace is not adequate. A useful hint is to look for the folder name turning red during the drag operation; this indicates that "dropping" there will deposit items into that folder.
Q. How do I get more information about my file transfer?
A. Within the Transfer Manager window, either double-click on the transfer's status line, or single-click on the "circle-i" button.
Q. How do I interrupt a transfer or other operation?
A. To interrupt file transfers, select the line of interest in the Transfer Manager dialog, then click the Abort button. To interrupt other types of operations, press the red Stop button. Note that some actions are not interruptible, and will be indicated by the Stop button being disabled (grayed out).
Q. How do I select more than one file or directory?
A. In the tabular display mode, click on the first item of interest and Control-click on any additional items you wish to select. If you want to select contiguous items, click in the white space following a file or directory name and sweep the cursor up, down, or to the side to select a range. Selections in the detailed display mode are similar, but you can also extend a selection by Shift-clicking. If you want to select all items, choose "Select All" from the Select menu. (On a Mac, use the Command key instead of Control.)
Q. How do I synchronize two directories?
A. Hopper can now perform rsync-style directory synchronization operations. This uses rsync when it is available, and uses built-in logic for doing synchronize operations involving hosts that do not support rsync. For example, suppose we want to synchronize from directory DirA on host HostA to directory DirB on host HostB. To do this, first open directory windows for both HostA and HostB. Select DirA on HostA and drag it atop the directory icon or name for DirB on HostB. From the resulting pop-up menu, choose "Synchronize this Directory".
Before a synchronization operation is actually invoked, a tabbed dialog is displayed that describes the operation and allows you to set various options. Take a look at the Function tab, which is of particular interest. Note that you can perform a dry run.
Once a synchronization operation is underway, it is run in the background as a file transfer job. Use the Transfer Manager window to keep track of the synchronization job. As with any file transfer job, the Transfer Manager allows you to display a file transfer monitor window so you can view details of the transfer (and this is where you can view results of dry runs).
Q. What is the "C" button? What do you mean by "caching"?
A. The "C" button toggles directory caching. When active, a directory's content information is cached in memory for use during that session. This removes the need for a directory listing operation the next time a previously visited directory is displayed. This can be useful when connected to very slow remote systems. Even when caching is on, Hopper will get a new listing if it knows the cached directory changed. When caching is off, a fresh directory listing is generated each time a directory is visited.
Q. How do I tell Hopper to pass command-line flags to ssh, scp, or sftp?
A. Simply enter the command-line flags following the executable names in the Connection -> SSH preferences. For instance, if you're using certain versions of OpenSSH clients and want to force protocol-1 to be used, set the ssh path preference to "ssh -1".
Q. How should I connect to storage (HPSS)?
A. Using the "Connect to Storage" item from the Connect menu will use the best available technique for connecting to storage from the given host. Users can also use the "Connect to Remote" item. On LLNL production hosts, there are four choices for connecting to storage: HPSS, NFT, HSI, and FTP. The HPSS option provides a full-featured interface to storage. This includes changing permissions and group values. Transfers from production machines use parallel FTP. The NFT option is comparable to the NFT command-line client; performance is slower than HPSS for some operations, but transfers are parallel and persistent. The FTP option uses standard, serial FTP transfers, and offers a reduced set of functionality. The HSI option is comparable to the HSI command-line client.
Q. Why do my xterm windows go away when I exit Hopper?
A. By default, xterm windows launched via Hopper will persist after Hopper exits. However, this behavior can be changed via the "Preserve remote xterms" item in the Operations -> Launch Xterm preferences. If your xterms are going away against your wishes, restore this preference to the factory setting.
Q. How can I make Hopper correctly display or operate on directory names which contain Unicode characters?
A. On some Unix-based systems, setting an environment variable that defines localization will allow Hopper to display and/or operate on Latin Unicode paths correctly. Try setting LC_CTYPE to "en_US.UTF-8" before launching Hopper. This should work on many Unix systems, but on some it will be ignored, and on others it may slow Hopper interactivity.
Q. How can I find out what's going on behind the scenes with Hopper?
A. Hopper is a very complex application and often manages multiple connections and dozens of threads. The "Audit Manager" window (via the Windows menu) lets you see what Hopper is really doing. The Actions tab shows all of the high-level commands executed by Hopper, including transfers and other directory operations. The Connections tab shows all of the connections being managed by Hopper, including active and inactive (cached) ones. The Passwords tab shows information related to password prompts -- e.g., who is asking for a password and why.
Q. Something went wrong. Where do I find the error messages?
A. If an error occurs during a non-transfer operation, Hopper's status field will say "ERROR -- CLICK HERE FOR DETAILS". Clicking on that message will bring up the diagnostics window, which you can get to at any time via the Windows menu. If an error occurs during a file transfer, information is displayed in the Transfer Manager window, also available via the Windows menu. If an error occurs during a search, see the Search Manager window.
Q. Why can't Hopper connect to [host]?
A. First, make sure that the host is up and accepting connections by trying to connect manually. (Use the same option you tried in Hopper -- e.g., if you tried to open an ssh connection in Hopper, try running "ssh host" from a shell window.) If the manual connection works, check your "Connection Types" preferences to make sure everything is correct. For failed ftp connections, try changing the "passive mode" preference value. If you're still stuck, contact the LC Hotline.
Q. I'm getting fewer (or more) password prompts than I expected. Why?
A. Hopper does a lot of work to reduce the password prompts you see; freed connections are cached and re-used and authentication information is stored during the session and re-used whenever possible. However, Hopper is a highly concurrent application; you can start one transfer, manipulate files in the source or sink window during the transfer, and even start additional transfers. The price for this flexibility is that multiple connections are required. In most cases, you'll only have to enter your password for a given host just once. In the worst case, however, a password prompt is required for each connection.
On systems that support it, you can reduce password prompts by setting up SSH keys. There are some cases when using scp transfers that you will have to authenticate frequently, possibly once per file moved. For this reason we set the default secure transfer mode to sftp, and encourage its use.
Q. Hopper sometimes slows down if I run it on a remote host and display it locally. Why?
A. Hopper performs well on local and most remote systems. Slowdowns are usually caused by the remote host being heavily loaded (check the load average reported by the Unix 'uptime' command.) Also, we've seen some versions of SSH that perform poorly when tunneling X11. Java applications are particularly sensitive to SSH's implementation of X tunneling. OpenSSH versions other than 3.6p1 seem to have the best X tunneling performance. One way to speed up X11 applications is to disable X tunneling, such as with the xauth application. Note that when SSH X tunneling is bypassed, any passwords entered via Hopper will be sent in the clear over the network.
Q. Double-clicking on the Hopper jar file on my Windows system does not launch Hopper, even though I have the proper Java installed. Why not?
A. Windows systems usually have an association between jar files and java, such that double-clicking on a jar file launches java. This is not always set up, though. For this reason we now distribute Hopper as a self-contained .exe file for Windows to avoid these potential issues. The .exe file also verifies that a recent-enough version of java is available.
Q. Why doesn't the "Launch Xterm" button work on the Mac?
A. It will work, but requires some set-up. There are three things to check. The first needs to be done each time you run Hopper. The last two need only be done once.
- Make sure the "X Window System" application (usually at /Applications/Utilities/X11) is running.
- Make sure ssh is set up to forward X11. This can be done by adding a command-line option in the Hopper preference for ssh (Preferences -> Connections -> SSH -> Path of ssh). Set this to "ssh -Y".
- Make sure your DISPLAY environment variable is set by running this command in a terminal window:
defaults write ~/.MacOSX/environment "DISPLAY" ":0.0"
After executing this command, log out and log back in again, and Hopper should be able to launch local and remote Xterm windows.
- Added a "structured copy" option that preserves the structure of the directory tree containing the files to be copied.
- Added a "Connect to Prod Host" menu item to the "Connect" menu to conveniently connect to clusters and specific login nodes.
- Introduced a new, minimal transfer status display to the bottom of the directory windows. Now one can see at a glance how many active, failed, and successful jobs have been launched.
- When run on LC production hosts, Hopper now automatically adopts your other jobs running in the same LC zone (CZ, RZ, SCF). This job information is available both in the minimal and the full transfer status displays.
- Directory window operations are now concurrent. Operations that previously tied up the entire user interface now tie up just a single directory window.
- Significantly enhanced the file transfer choice algorithm, which reduces the number of two-hop transfers when working with multiple zones and/or multiple users.
- Made detailed file transfer window more memory efficient.
- Decreased memory footprint of "Show Clipboard" dialog.
- Decreased memory footprint of audit dialog.
- New automatic directory refresh capability.
- Added a ping of prospective Hopper server hosts for availability before attempting to connect.
- Can now make HSI connections to LANL and Sandia HPSS storage systems (hpss.lanl.gov and smss.sandia.gov) on the SCF.
- Modified server timeout behavior so that an attached client does not prevent server termination after a prescribed amount of server inactivity.
- Fixed issue preventing TAR file updates from completing successfully.
- Chopper: added aggregated copy and multi-volume copy commands.
- Added ability to filter file transfer logs in order to view error messages only.
- Fixed issue preventing NFT file transfers from being aborted properly.
- Fixed memory leak that could crash long-running Hopper servers.
- Significantly enhanced support for Enclave (multi-zone and multi-user) operations.
- Improved speed of status database when operating on large number of files.
- Added more per-user disk usage views. View usage by directory, by users, or for a specific user per directory.
- Added ability to convert format of text files.
- Improved performance of client/server communications.
- Reduced password prompts for desktop operations involving RZstage.
- Reduced password prompts for access to RZFIS.
- Enhanced disk usage help information.
- Reduced amount of memory used for job descriptions.
- Added preference for disabling use of POD.
- Added ability to view disk usage (DU) for multiple users, such as for a shared project directory.
- Fixed problem that could result in excessive memory use when creating an HTAR archive.
- Fixed DU error that could result in items not being counted during a scan.
- Fixed DU error that affected scans run on production machines if the scans were started from the root of any LC shared directory.
- Fixed Windows-specific DU error that affected scans that were started from the Windows root directory.
- Removed retired hosts and added new hosts to site file.
- Added Multi-Volume Archive help information.
- Fix for error that occurs when transferring files to/from some non-LC FTP servers.
- Added workarounds to the file transfer logic in order to accommodate an FTP client that hardwires user names when connecting to storage.
- Fixed problems with file transfer logic that occurred when initiating transfers from the EN between CZ and RZ/IZ.
- Fixed race conditions between threads that could occur when parsing directory list output.
- Can now handle an unexpected date/time format in the directory listing returned by a Windows-based FTP server.
- Fixed an array-out-of-bounds exception caused by an update to the directory window in detailed view made by a thread other than the event-dispatch thread.
- Improved management of jobs in order to correctly handle multiple user names per user.
- Added multi-volume archive operation; a convenient way to transfer large directories to storage.
- Better usage of existing host connections. Previously, Hopper required separate connections for the directory window and for each simultaneous transfer involving that host. Now, a single connection can be used for both a directory window and a single transfer as long as they don't both need to use the connection simultaneously. This can result in fewer password prompts and better interactivity.
- Many usability improvements related to HPC Enclave, including fewer (and better defined) password prompts, particularly when connecting to RZFIS and RZstage from desktop machines.
- Transfers with HSI now make use of HSI's multi-file parallelism.
- It now takes significantly less memory to internally represent directory listings.
- Better file locking code for the file-transfer status database.
- Removed limit on size of search results.
- Numerous bug fixes, optimizations, and general improvements.
- [Mac] Removed dependency on PPC version of JavaApplicationStub that prevented start-up on OS X 10.7.
- Fixed issue where output from some operations could leave directory window in non-functioning state.
- Disabled the use of SSH ControlMaster for Hopper SSH-related connections that do not need it.
- Changed site file so CZ machines are not treated as production machines within the RZ .
- Support for RZstage host, including ControlMaster connections via JSFTP to minimize authentications.
- Added retry logic when some status database operations fail due to file system glitches.
- No longer allow NFT transfers when source and sink user names are not the same.
- Enhancements in support of HPC Enclave (CZ/RZ).
- New Multi-Volume copy operation automatically breaks very large transfers into manageable TAR/HTAR archives.
- New history editor function allows users to remove/arrange various cached values.
- Converted message exchange from XML to serialized objects.
- Allow aggregated copy operation in more cases (e.g., machine to machine).
- New autosave preferences, configuration information, history, etc.
- Now use built-in splash screen for faster startup.
- Improved memory usage for directory parsing.
- Improved error and status reporting for aggregated copy, sync, and TAR/HTAR.
- Removed inverse video for all directory views.
- "Show Clipboard" dialog now resizes correctly.
- No longer encounter errors when operating on many search hits.
- Fixed bug in Connect to Remote that could inappropriately disable Connect button.
- User name now restricted to login name when connecting with HPSS, HSI, NFT.
- Updated to latest HPSS shared library.
- Various improvements and bug fixes in handling of TAR and HTAR files with unusual contents.
- Fixed bug causing non-optimal selection of server host. Result to the user could be unexpected password prompt.
- 2.0.0 (b30-45)
- 2.0.0 (b1-29)
- Extensive improvements to job handling, including ability to monitor jobs from multiple clients, nearly instantaneous job adoption, and the ability to easily resubmit previous jobs. This is accessible by clicking on the "More Details" button in the Transfer Manager dialog.
- "Copy Here (with options)" added to drop menu. This contains some of the less frequently used copy operations.
- Added simple synchronization option with copy (via "Copy with Options").
- Added ability to transfer remote TAR and HTAR files to a desktop and automatically expand them.
- Added copy-with-rename operation (via "Copy with Options").
- Added duplicate operation (via the right mouse menu).
- Added checksum option to the synchronize operation.
- Optimized handling of diagnostics messages between client and server.
- Added built-in viewer capability to the Content Mode's preview pane. Now can preview text and image files directly.
- Added real text preview to text icons in Content Mode.
- Improved robustness of Java FTP authentication.
- Improved Endeavor job handling so that jobs are queued immediately.
- Can now reconnect to Endeavor server after network interruptions or server restarts.
- Improved memory use when viewing contents of TAR or HTAR file. Now can open arbitrarily sized archives.
- Added "Copy if needed" option to copy only if the source file was changed or added.
- Fixed problem causing bad password to be re-used in certain circumstances.
- DuManager no longer kills server if can't create /nfs/tmp2 subdirectory.
- Fixed display problem with Terminal Manager on Mac.
- Fixed handling of asterisks in file names with FTP file system.
- Fixed problems handling HTAR files with bad indices.
- Fixed problem on Windows when using SCP with the preserve timestamp option.
- Opening HTAR when storage is down no longer throws an exception.
- Now display proper size for files >4 GB on 32-bit systems.
- Searching by permission or by date no longer throws exception.
- Reordering columns in transfer manager dialog no longer throws exception.
- Fixed problem preventing aborting of some aggregated copies.
- Connection problems between client and server no longer cause server to exit.
- Aggregated copy no longer throws exception if source files are deleted during process.
- Clicking on search hit with 1000s of lines of context no longer hangs.
- Displaying du info for empty dir in tar/htar archive no longer throws exception.
- Fixed sync issue with multiple "cache" or "hidden files" toolbar buttons.
- Added toolbar editor functionality. Now you can personalize the toolbar by choosing the button operators as well as their locations. Several new operations are now available via the toolbar.
- Added two new buttons to the default toolbar: "search" and "new window".
- Added new copy option called "Copy with htar expansion", available when transferring from storage to a production host. This acts like a normal copy except any HTAR archives will automatically be expanded in place. If you want to completely expand an HTAR archive, just drag the HTAR file from storage and choose "Copy with htar expansion". This will cause the contents of the file to be expanded in the directory on which you dropped it.
- Added support for "Give" and "Take" on LC production hosts. To give files to another user, highlight the file(s) and use the "Give Files..." option from the right-mouse menu. To take files another user has given you, bring up the right-mouse menu when over a directory icon or white space in the Hopper directory window and choose "Take Files...".
- Added "sort by suffix" option in Hopper's tabular view.
- Now can open arbitrarily large HTAR archive.
- Now have ability to manually edit a "launch" command just prior to launching it on selected items.
- Increased max number of unique directories displayed in the disk-usage bar-chart view.
- When viewing a remote file, now limit the file size retrieved to a local temp directory.
- When using the backup feature of synchronize, backup dirs are no longer created if no files are backed up.
- Fixed problem when connecting to Mac from Windows caused by changes in OS X 10.6 SSH behavior.
- Fixed problem caused by changes to the TAR utility under Mac OS X 10.6.
- Added Disk Usage view. Click "pie chart" icon in toolbar to open the disk usage pane.
- Various minor bug fixes.
- Windows: Fixed tar and copy related problems dealing with drive-level folders.
- Windows: Now support direct SFTP connections and transfers.
- Fixed issue caused by putting top-level directory in clipboard.
- Now check for existence before creating an htar/tar/jar/zip archive.
- Now display more error output when a user-defined Launch Command fails to run.
- Sped up searches in built-in viewer.
- Fixed problem dealing with embedded spaces in directory names with FTP connections on Mac OS X 10.6.
- Now remember column configurations between sessions when in detailed (tree) view mode. This includes column sequence and widths.
- Added Aggregate Copy feature for efficiently storing huge directory trees.
- Added sort-by-type option for content view.
- Added timestamp-preserving copies to storage.
- Removed file size limit for built-in viewer. Now file size is limited by temporary disk space available for index file.
- Default for emptying Hopper storage trash can is now 30 days. Items older than 30 days old will be removed the next time Hopper is run.
- Now handle case where directory from which Hopper was started is deleted.
- Now prevent Endeavor connections to non-suitable hosts.
- Now check for missing DISPLAY variable in Hopper start-up script.
- Now display proper file count when transferring files with HSI.
- Now can connect to remote servers that use old /bin/sh.
- Fixed problem causing remote tar file creation on UNIX hosts to fail in some circumstances.
- Fixed issues related to making FTP transfer of top-level folder on Windows.
- Fixed rare problem causing active search and transfer jobs to hang after adoption.
- Now remember column configurations between sessions when in detailed (tree) view mode. This includes column sequence and widths.
- Fixed problem causing some two-hop transfers to fail if a storage proxy host was being used.
- Now can connect with ssh to MacOS X 10.6 system.
- Simple recursive copy no longer terminates if it encounters a bad file.
- Worked around problem with IBM java causing spurious end-of-stream errors.
- Adopting huge jobs no longer hangs up.
- No longer get spurious second abort request following an initial abort.
- Fixed array out-of-bounds problem when dealing with single character paths in Windows.
- Fixed issue from desktop that would report "copy operation not supported" for some 3rd party transfers.
- New directory window view, "Content View". Files are represented with icons based on their type (source files, binary, PDF, etc.). Image files are represented with thumbnails.
- Added automatic retry logic to FileSync. If there are failures (with either rsync or built-in) detected during a FileSync, then the entire synchronization will be repeated in an attempt to pick up anything that was missed due to the failures.
- Added code to FileSync to prevent a user from performing simultaneous identical synchronizations.
- A number of changes were made to HpssFileSystem and HpssThread to improve behavior when Hopper is unexpectedly disconnected from HPSS.
- Sync operation no longer creates zero-length files when underlying transfers fail in unusual way.
- Fixed problem causing rare, spurious NPE on Hopper start-up.
- Fixed various problems that might cause an exception when working with the Hopper help package.
- Starting two clients simultaneously no longer generates exceptions during subsequent operations.
- Exiting client while an abort operation is underway no longer generates an exception.
- Closing confirmation dialogs with the window manager, rather than clicking "yes" or "no", no longer generates an NPE.
- Now check that user has account before launching server on Slic node.
- Improved renewal of Kerberos credentials.
- Improved output from running built-in sync in dry-run mode.
- Fixed bug causing search function in the Help package to throw an exception.
- Fixed OOM during multiple built-in file syncs of large directories.
- Fixed exception caused by SFTP returning null link target.
- Fixed rare OverlappingFileLockException problem when adopting jobs.
- Fixed memory leak in client.
- Now print site file being used in client debug file.
- Now detect and recover from deletion of sshctlr executable.
- Improved error message when trying to overwrite a dir with a file by the same name (UNIX).
- Reworked logic in directory parser in ways that minimize memory usage.
- Fixed retry logic for FTP connections to servers that don't accept "ls -al".
- Aborting Unix operations now occurs more promptly.
- Fixed bug causing some Lustre dir listings to hang if another file
system was down.
- Made Endeavor disconnect improvements.
- Added ability to specify port number in chopper connection.
- Fixed bug causing built-in viewer to fail when file-size preference
was set to infinity.
- Added preference for emptying Hopper's HPSS trashcan
- Now better handle unexpected Endeavor (NFT) disconnections.
- Sync operations with htar archives as the source now behave properly.
- Add line number option to viewer invocation in Search window. (Now can start
up editor on exact line of a search hit.)
- Now renew kerberos credentials automatically.
- Now handle symlinks properly on Mac Leopard (10.5) systems.
- Built-in viewer now recognizes more types of images, particularly
newer jpeg formats.
- Fixed bug affecting some long-running indexed tar archive operations.
- Now treat .war and .ear files as jar files.
- Improved searches involving dates on Mac.
- Added built-in file comparison package for comparing and merging text files; user preference controls whether built-in or external comparison tool is used. Initiate comparison by dropping one file on top of another.
- Added complete Help package, accessible via both Help menu and Help icons on various dialogs.
- Added search capability to built-in viewer.
- Set timestamps immediately with sync when possible.
- Change default so dot (hidden) files not shown.
- Made preferences pane non-modal.
- Now include user name in password prompt dialog in more cases.
- Now included FTP-related transfer rate information in diagnostics.
- Improved default file viewers for Windows.
- Improved diagnostics for HPSS-related errors.
- Added HTAR extraction queue.
- Added HTAR creation queue.
- Added HTAR timeout to avoid indefinite hangs when HTAR fails.
- Simplified connection process for HPSS and FIS from desktops.
- Simplified Hopper trashcan naming conventions.
- Extended trashcan to work with HPSS regardless of connection type.
- Improved recovery from unexpected HPSS downtimes.
- Fixed out of memory error from adopting a huge job.
- Fixed problem preventing aborting of adopted jobs.
- Added "check for updates" and "provide feedback" entries to Help menu.
- Added more feedback during multi-hop transfers.
- Added feedback to "Adopting Jobs" dialog.
- Fixed numerous problems with remote tar creation.
- Now quickly abort two-hop transfers.
- Now promptly kill HTAR process when create operation is aborted.
- Now can search for a file name that is a number.
- Now properly mark search context lines that contain a colon.
- Now handle new 'h' identifier in latest GNU tar listings.
- No longer exit if directory in which server was started is deleted.
- New code dependencies now require at least Java 5 to run. Java 6 is preferred. Check version with "java -version".
- Fixed bug that prevented sync operations to ftp-connected hosts (e.g. between a desktop and LC storage) from functioning properly. This did not affect syncs between LC production hosts and storage.
- Now optimize transfers between production hosts and FIS.
- Now retry when experiencing HPSS connection problems.
- Now recover from unusual Endeavor disconnections.
- Significant expansion of synchronize capabilities, including new options to ignore existing sink files (i.e., only transfer missing files), to only consider files that are present on both the source and sink, or to only use the size when comparing files (i.e., ignore timestamp).
- Added size-only comparison for sync operation (i.e., ignore timestamp).
- Added sync option for only transferring missing files.
- Now ignore seconds in sync comparison if connection protocol doesn't support it.
- Added YY/MM/DD to job debug info.
- Added support for preserve-timestamp option to underlying copy mechanism.
- Added info icon buttons to job status dialogs.
- Now handle zero-length searchRuleSets, historyRev1 and hostmarks files.
- Now include date with start/end times in transfer monitor.
- Now can show actual seconds in long list if underlying protocol supports it.
- Improved performance of directory parsing logic.
- Fixed directory parsing issue on Mac Panther systems.
- Improved error messages caused by unexpected SFTP output.
- Improved robustness of terminal interaction code.
- COS now always set for large files written to storage from desktop.
- FTP and HSI child processes now always terminated if transfer aborted.
- Now can access search hits even when adopting job from another host.
- Syncs from storage to local host no longer transfer more than necessary.
- Server no longer exits if user prematurely terminates client.
- Server no longer exits if sync job aborted immediately.
- Server no longer exits if aborting an SFTP transfer.
- Now can use SFTP on Fedora Core systems.
- Bad SFTP connections no longer use CPU time.
- Strange SFTP progress messages no longer generate warnings.
- Various other bug fixes.
- Now support direct-from-tape htar extractions when reading small number of files from large archive. (Significantly faster.)
- Now support transparent use of storage proxy host such as oslic and cslic.
- "No locks available" NFS bug no longer causes permanent hang; server now recovers once locks are freed.
- Now support hiding of dot files when viewing archives.
- Improved diagnostic messages in log files.
- Minor changes to chmod GUI to reduce confusion.
- Numerous HSI reporting improvements.
- No longer have to scroll menu to see all search criteria options.
- Streamlined display of transfer information in client to remove memory issues.
- Various other bug fixes.
- Completely revamped and simplified search interface, with new ability to graphically construct search queries. File searches now support date and age options, in addition to name, permissions, owner, group, size, type, and content. Content searches now support optional lines of context.
- Streamlined job adoption process to use significantly less time and memory.
- Sped-up and simplified all SSH-based logic as well as invocation of remote commands.
- Added full support for HSI utility, including connections to storage and transfers to/from storage.
- Resolved server out-of-memory errors caused by huge transfer and search jobs.
- Resolved client out-of-memory errors caused by huge transfer and search jobs.
- Restored ability to scroll through search hits using up/down arrow.
- Server now exits cleanly when a thread encounters a fatal error.
- Improved detection and handling of server failures in client. Now, if server dies all existing jobs will exit cleanly and new server will be started.
- Added auto-cleanup of Hopper-created log files, tmp directories, etc.
- No longer can copy a directory onto itself or descendant.
- Removed occasional extra password prompt when running rsync on SCF.
- Added trashcan for deletes in HPSS from HSI connections.
- Added COS setting for streaming tar files created in HPSS.
- Improved reporting to client of server errors.
- Fixed problem causing rsync between production and non-production system to fail.
- Fixed rare but recurring "no locks available" error in server.
- Fixed HTAR extraction problem when archive was created using "./" notation.
- Various other bug fixes.
- Better handling of dirs with spaces during archive extraction (cmdwrapper).
- Verify existence of status db lockfile before using.
- Improved handling of archives containing odd characters in name.
- Fixed problem with using ' <' operator in "find" mode.
- Now print unexpected statusdb server errors to log file.
- Optimized server choice when adopting jobs.
- Various bug fixes in: indexed tar archive creation, ftp handling of unusual characters, Endeavor server choice.
- Revamped architecture based on client/server model that (1) allows client to disconnect while server continues to perform operations and (2) significantly reduces the number of two-hop transfers involving production hosts.
- Added "favorites" capability for choosing favorite connections and directories. With the associated autoconnect option users can do things such as automatically open a window to a certain directory in storage when Hopper starts up.
- New "smart" mode for automatically selecting transfer technology. This mode, the new default, will be optimized for both speed and minimization of authentications based on knowledge of the source and sink host.
- Added ability to operate on search hits. Users can drag and drop search hit items or choose operations from a pop-up menu including copying to clipboard, deleting, changing permissions, viewing, etc.
- New ways to perform data aggregation during a transfer: users can create a tar or indexed tar file on a remote host from local data, even when running on desktop systems. This aggregating style of transfer is up to 10 times faster than copying files individually with FTP.
- Support for comparing any two files, even if on different hosts. Initial version supports use of external diff tools, such as xdiff, on Unix hosts; future version will also have a built-in diff tool for use on any platform.
- Added pre-authentication logic to guarantee that a job has all necessary information before allowing Hopper client to exit.
- Autoscrolling enabled when extending selections in directory windows.
- Comprehensive audit capability.
- Simplified process for getting into 'rename' mode. Now can select file and hit RETURN.
- Added "clear cache" shortcut to the Connect menu.
- Added logic to validate a cached connection before re-using it.
- Improved and simplified preference layout and internal representation.
- Added client-side logging.
- Added support in the hopper wrapper script for executing arbitrary versions of Hopper using the -v flag.
- Added support for <,>,<=, and >= operators when examining file size in the "find" option of the search operation.
- Improved error message when trying to perform a GNU tar operation on a machine that does not have GNU tar.
- Added symlink support for FTP connections where client and server both support it.
- Added support for following symlinks when creating tar file.
- Added support for HSI utility.
- Added ability to change permissions and group through jftp connections (e.g., on Windows).
- Added support for dual-copy class of service for writes to HPSS.
- Reduced memory use when handling large transfer and search progress messages.
- Fixed synchronize bug causing incorrect modification time for files on Windows.
- Fixed synchronize bug causing created directories to have wrong mod time.
- Fixed bug causing xdiff to be launched without appdefaults file (Java bug).
- Modified LDAP connection code to retry in case of slow ssl handshake.
- Fixed bug causing Hopper window to become unusable if underlying working directory was deleted.
- Fixed bug preventing zip archive extraction on Windows.
- Fixed failover issues with HPSS connections.
- Force garbage collection when clicking on memory usage icon.
- Sped up job adoption from remote server.
- Revamped server start-up logic to use command-line ssh if available.
- Sped up htar extractions.
- No longer select a directory when expanding it in tree view mode.
- Allow users to choose transfer method for synchronize operation.
- Prettified copy choice menu.
- Worked around ftp problem caused by filenames with leading dashes.
- Fixed hang caused by transferring zero-length file with scp.
- Fixed bug causing server to not terminate after completing last job.
- Fixed bug causing error messages during deletion from zip archive.
- Fixed bug causing synchronize-with-backup option to sometimes fail on Windows.
- Fixed problem when aborting tar file operations.
- Fixed problem caused when using a timed-out Endeavor connection.
- Fixed jftp exceptions.
- All client actions are now logged in the audit window.
- Error status now cleared when status button clicked.
- Significantly sped up GUI updates when reporting large amounts of information (e.g., lots of paths).
- Defined copy-choice sequences via the Site file to allow easy tweaking during deployment.
- Changed file name escape method for FTP connections to deal better with unusual characters.
- Fixed bug preventing searches of HPSS to properly detect tar and htar files.
- Fixed bug causing timeout when using FTP non-passive mode to connect to a host that requires passive mode.
- Fixed bug in the "locate" option of the search operation that caused file content searches to have incomplete labels if only one match was found.
- Fixed bug that could cause incorrect file count to be reported during synchronize operation.
- Fixed bug when using rsync on remote hosts with --include or --exclude option.
- Fixed bug causing NPE if running htar at a site without htar support.
- Fixed bug in EndeavorFileSystem preventing creation of relative symlinks.
- Fixed bug in FtpFileSystem triggered by unusual filenames.
- Fixed bug causing spurious error message when deleting > 10 files.
- Fixed bug causing connection build-up when using built-in sync op.
- Fixed bug causing some remote htar operations to fail when run on Mac.
- Fixed bugs causing abort of delete operations to leave Hopper in bad state.
- Fixed bug causing FTP connection error from some Solaris systems.
- Fixed bug causing comparison of two remote files of same name to fail.
- Fixed bug causing stack trace when saving output from sync log.
- Fixed bug that prevented audit manager from being displayed under Gnome after being minimized.
- Worked around Swing bug causing occasional stack traces on AIX during startup.
- Windows: Mod to accommodate new variation on listing output [caused hang].
- Mac: Fix to handle OS X directory lists containing ACL information.
- Site configuration modifications.
- Minor mods to support peloton systems.
- Site configuration modifications.
- Bug fix: ignore bad certificates when checking for updates.
- Cache HPSS-related connections (minor speed improvement).
- Bug fixes: incorrect behavior for synchronize operations due to incorrect initial settings; result was that some files may be transferred unnecessarily.
- Fixed problem that caused occasional ssh connection failures on PC.
- Now completely exit htar client if user aborts htar operation.
- Added large-temp and small-temp concept to minimize problems caused by scratch directory going offline.
- Ability to save contents of file transfer dialog.
- Windows: fixed loop when trying to ssh to host that rejects ssh.
- Site configuration modifications.
- Fixed several sync problems, including ones that prevented rsync from working, and one that caused extra connections to be left behind.
- Cleaned up splash screen.
- Simpler interface for making connections.
- Numerous performance improvements with both local and remote connections.
- Updated interface with improved icons.
- Much faster opening of existing htar files (up to 2x faster).
- More detailed status information during tar/htar operations.
- Ability to choose file transfer type dynamically.
- Simplified preferences for selecting default transfer type. Have a choice of: "best", "best secure", and "display all choices".
- Ability to view same directory in multiple windows.
- Automatic notification when a new version of Hopper is available.
- Audit feature which tracks operations Hopper performs, the connections it has open, etc.
- Significant speed-up when moving large numbers of files.
- Support for 64-bit AMD systems.
- Mac OS X 10.4 (Tiger) optimizations.
- Faster ssh-based transfers when running on Windows.
- Generalized Windows path handling; now supports UNC Windows paths such as "\\host\path".
- Support for DSA copies.
- Support for connecting to esoteric ftp servers.
- Ability to open larger files with built-in viewer.
- New memory usage display.
- Complete overhaul of internal architecture in preparation for client-server release.
- Various bug fixes and optimizations.
- Added support for running on Mac OS X 10.4 (Tiger).
- Fixed problem caused by ptys on OSF1, Tiger, and Panther.
- Fixed problem when find returns blank lines.
- Fixed problem when opening a tar file that doesn't have user or group info.
- Fixed problem caused by unusual dir output in Windows.
- Fixed termination hang-up caused by using Java 5 on AIX.
- Fixed bug that prevented "drop menu" (following drag and drop) from appearing in certain circumstances.
- Fixed problem when opening or extracting from certain tar files.
- Fixed confusing prompts when doing synchronize operation.
- Fixed problem with htar extract caused by different local and remote user names.
hopper − Graphically manage and transfer files
hopper [-beta] [-debug] [-help] [directory]
−beta: Run the most recent beta version of Hopper. This will have the latest well-tested features, but still may contain bugs.
-debug: Turn on the debug property, causing additional information to be printed to the console.
-help: Prints a brief help message.
The following operand is supported:
directory: Specifies the initial directory to be viewed within Hopper. If none is provided, the current directory is used.
To start Hopper and view your current directory: hopper
To start Hopper and initially view the /tmp directory: hopper /tmp
Connecting and Transferring
Upon start-up, Hopper will show a directory listing for the current directory on the machine on which Hopper is running. To connect to "storage", use the "Connect" menu and choose "Connect to Storage (HPSS)". This will bring up a listing of your home directory on storage. Hopper can also connect to any other machines available via SSH and FTP. These options are also accessed from the "Connect" menu.
Hopper has an easy-to-use drag-and-drop mode which makes it simple to graphically perform data operations. After selecting entries and dragging to the desired destination, a menu will pop-up from which you can choose an operation to perform -- copy, move, create htar file, create tar file, synchronize, and so on.
Files can be transferred by dragging and dropping the file/directory icons from one window to another. The cursor icon will change to a "smiley face" when it is over an allowed drop position.
Multiple files can be selected by dragging with the left mouse button held down. A selection can be extended by first clicking on one item and then Shift-clicking on another. Additional items can be selected by clicking on an item with the Control key held down. (Use the Command key on a Mac.) Also, typing the first few characters of a file's name will cause that file to be selected.
Hopper can perform a number of operations on individual files or groups of files. These operations are invoked via the right mouse menu -- click the right mouse button over a file or directory, and you’ll see a set of operations you can perform on that item. If you have a one-button mouse, this menu can be activated by clicking while holding down the Control or Command button. The operations include:
Check Access: This reports the users and groups that can access a given file. This is useful for checking if a particular person or group has access to one of your files. It takes into consideration the permissions of all enclosing directories. It is selected by performing a "right click" on the file of interest and choosing "Check Access" from the resultant pop-up menu.
Change Permissions: This brings up a dialog box that can be used to change permissions on selected items. It is selected by performing a "right click" on the file(s) of interest and choosing "Change Permissions" from the resultant pop-up menu. There is a "recursive" check box that when selected will cause the permissions to be changed recursively for all files contained in any selected directories.
Change Owner/Group: This brings up a dialog that can be used to change owner and group information on selected items. It is selected by performing a "right click" on the file(s) of interest and choosing "Change Owner/Group" from the resultant pop-up menu.
Delete: You can delete files by either (1) selecting the files to delete and then clicking on the trash can icon in the toolbar, or (2) select the files and then choose Delete from the right mouse menu. By default you will be prompted for confirmation before any files are removed.
Make Directory: To create a new directory or folder click the "New Folder" icon in the toolbar. After you supply the name, the new directory will be created in the current directory.
Rename: This highlights the selected file or entry’s name, allowing you to rename it. Press Return to complete the rename, or Escape to cancel out of it. It is selected by performing a "right click" on the file(s) of interest and choosing "Rename" from the resultant pop-up menu.
View: To view a local or remote file, double-click on the file. For remote files, this will transfer the file to a temporary directory on the local host for viewing. For local files, viewing is done in place. Several types of files can be viewed using Hopper's built-in viewer; these include jpeg, gif, and png image files, plus text and raw binary files. The viewer is chosen based on the file suffix (e.g, ".jpg"). Alternative viewers can be assigned via the preferences: Preferences/Operations/File Viewer. Files can also be viewed by performing a "right click" on the file of interest and choosing "View With" from the pop-up menu.
Reading and Creating HTAR and TAR Files
Hopper can open and create htar and tar archives. In directory listings, htar archives will be shown with an "H in a circle" icon. You can double-click on one of these icons to show the directory listing for the htar archive. Files can be copied out of an htar directory just like any other directory. Files cannot be copied into an htar directory, however, as htar archives are read-only. Tar archives appear with a "T in a circle" icon and are operated on in the same fashion as htar archives.
Htar and tar files can be easily created using the drag-and-drop interface. To create an htar file, just select the files you want to make an archive from, and drag them to a storage or production host window. Choose "Make HTAR Archive Here" from the menu to complete the operation.
Hopper can now perform rsync-style directory synchronization operations. For example, suppose we want to synchronize from directory DirA on host HostA to directory DirB on host HostB. There are several ways to do this, but for clarity we will show you the most straightforward way. First, open directory windows for both HostA and HostB. Select DirA on HostA and drag it atop the directory icon or name for DirB on HostB. From the resulting pop-up menu, choose "Synchronize this Directory".
Before a synchronization operation is actually invoked, a tabbed dialog is displayed that describes the operation and allows you to set various options. Take a look at the Function tab, which is of particular interest. Note that you can perform a dry run.
Once a synchronization operation is underway, it is run in the background as a file transfer job. Use the Transfer Manager window to keep track of the synchronization job. As with any file transfer job, the Transfer Manager allows you to display a file transfer monitor window so you can view details of the transfer (and this is where you can view results of dry runs). A couple of points worth noting:
When syncing to HPSS, only the size and timestamp of a file will be used to determine if it needs updating -- the actual content of the file is not examined.
The "create an exact mirror" option can cause files on the sink to be deleted, which can be a disaster if the sink directory is chosen incorrectly. There are "dry run" and "backup" options available that can reduce this risk.
Hopper now has a clipboard mechanism that can be used to initiate various file operations. The clipboard is accessed via (1) the Clipboard menu in the main menu bar, and (2) items in an entry's pop-up menu (use the right mouse button over an entry to pop up this menu). Clipboard-based operations are initiated in two steps. First, source items are put into the clipboard. Second, an operation is selected in the context of a destination directory. Note that there are menu items for viewing and clearing the contents of the clipboard.
The Hopper clipboard mechanism and associated keyboard shortcuts make it possible to do "dragless" copies. Select the files you wish to copy, and put them on the clipboard by typing Control-C (there are also menu options for adding files to the clipboard.) Pressing Control-V in the destination window will "paste" the files from the clipboard, thus initiating the copy operation. (Note that on the Mac one uses the Command (Apple) key, rather than Control.)
- Password handling is a strong point of Hopper. It contains logic to reduce the total number of password prompts you receive. However, there are circumstances where you may be asked to enter a password for a given host more than once. This is due to the fact that transfer operations occur in separate threads, and each thread requires its own connection.
- When connected to FIS with ftp, dot files do not show up.
- Color preview area does not show up under JDK 1.4.2. (Okay with JDK 1.4.1.)
- SCP transfer of several thousand or more files fails. (SFTP okay.)
- Trouble doing gets with OpenSSH sftp 3.7p1. Okay with 3.8.1 and 3.6.
Mac OS X Issues
- Pre-Panther only -- ssh connections sometimes hang up.
- When connecting to a Mac with FTP, listing "/" fails.
This section has moved to a separate user guide, "Hopper Help."
The only requirement for running Hopper on an LC production host is that your local system have the ability to display X windows from remote hosts. To run Hopper, simply type: hopper. LLNL users can download Hopper for their local systems from:
To run Hopper on your local system, follow these instructions:
- Check Java—Make sure you have Java version 1.5 or later installed. To check the java version number, run "java -version" from a terminal window.
- Download Appropriate File(s)—(from any LC production host)
- Complete the Installation
|Mac||Double click on the dmg file to open the volume, then drag the Hopper application to the desired location on your hard drive. Double click on the Hopper icon to launch Hopper.|
|PC||Simply double-click on the exe file to launch Hopper. No installation is necessary.|
|Unix||Rename the hopper.local file "hopper." This is a special startup script for use on local Unix systems. This is what you will run to invoke hopper.
Edit the hopper script so that it points to your copy of hopper.jar by changing the first occurrence of "HOPPER_JAR_FILE." Alternatively, you can set the environment variable HOPPER_JAR_FILE to point to the jar file location. You will also need to edit the hopper script to point to the Java executable on your system by changing the first occurrence of "HOPPER_JAVA." Alternatively, you can set the environment variable HOPPER_JAVA to point to the java executable location.
Basic usage information is available in the man page.