Directory Structure and Properties

This section shows how the important public directories (and their underlying file systems) are organized on LC machines. LC production Linux clusters differ in size, chipset, and available switch (interconnect), which can affect the scale and parallelism of applications for which each Linux cluster is best suited. See LC's Compute Platforms Web page for configuration details of the LC Linux clusters. Most systems run TOSS, the Tri-Lab Operating System Stack, a modified version of Red Hat Linux built for supercomputing at LLNL, and have the same basic file system hierarchy and public directory properties. 

Each Linux cluster has a public directory structure as described below. 

/tmp

A link to /var/tmp (for system use).

/usr/tmp

A link to /var/tmp (for general use).

/var/tmp

Temporary storage space for system activity and, optionally, for user activity. On diskless clusters, /var/tmp uses real memory, and TOSS purges it immediately after every batch job ends.

/nfs/tmpn

Large temporary file systems (currently /nfs/tmp2) globally available and visible by all open Network File System (NFS)-mounted LC production machines. This eliminates the need for between-machine file transfers and simplifies batch job preparations. Create your own subdirectory here for your computational work (rather than run big jobs in your home directory). Files in this directory are subject to purge. See the File Purge Policy section for details.

/g

A file system of globally available ("common") home directories on highly reliable NFS-mounted RAID (redundant array of independent disks) disks on each OCF and SCF production system. Your child subdirectory here (/g/gnn/yourname) is your default arrival directory and contains your startup and run-control dot files, but it is limited in size.

/opt

Contains third-party tools and is accessed via the module comand. Some of the compilers that LLNL uses reside in /opt, as well as the default compilers used by LANL and Sandia. Versions of MVAPICH and Open MPI compiled with the various versions of the compilers also reside in /opt. The compilers and MPIs in /opt are kept updated to match what is available in /usr/local/tools

/p

A set of Lustre parallel file systems designed to accommodate large, intensive parallel I/O. Each Lustre file system (lscratch) has either a unique one-letter (for OCF systems) or one-digit (for SCF systems) identifier. See the Parallel File Systems section for details.

/usr/apps

A link to /usr/gapps/$SYS_TYPE. See /usr/gapps below.

/usr/gapps

A file system of globally available common code-management directories on NFS-mounted RAID disks on each machine similar to the /g common home directories. The directory contains some important non-commercial shared application codes and tools, such as ALE3D, BASIS, HYDRA, and Python. See the /usr/gapps Web page for details about the /usr/gapps file system.

/usr/bin

Contains hundreds of standard UNIX software tools (or their Linux counterparts) along with the C and Fortran compilers.

/usr/local

Contains some important commercial shared tools, such as TotalView, Valgrind, mpiP, and the PathScale compilers.

/usr/workspace/ws[a,b]

A file system containing 1TB directories for every LC user and UNIX group.  This file system is not backed up, but does have one week of snapshots.