System Status and Configuration Information

• Before you attempt to run your parallel application, it is important to know a few details about the way the system is configured. This is especially true at LC where every system is configured differently and where things change frequently.
• It is also useful to know the status of the machines you intend on using. Are they available or down for maintenance?
• System configuration and status information for all LC systems is readily available from the LC Homepage and the MyLC Portal.

System Configuration Information

• LC Homepage: hpc.llnl.gov ==> Hardware ==> Compute Platforms
  • Direct link: hpc.llnl.gov/hardware/compute-platforms
  • All production systems appear in a summary table showing basic hardware information.
  • Clicking on a machine's name will take you to a page of detailed hardware and configuration information for that machine.
• MyLC Portal: mylc.llnl.gov
  • Click on a machine name in the "machine status" portlet, or the "my accounts" portlet.
  • Then select the "details", "topology" and/or "job limits" tabs for detailed hardware and configuration information.
• LC Tutorials: Located on the LC Homepage under the "Documentation" menu.
  • Direct link: hpc.llnl.gov/documentation/tutorials
    • LC Systems Summary: hpc.llnl.gov/sites/default/files/LC-systems-summary.pdf. Concise 1-page summary of LC production systems.

System Configuration Commands

• After logging into a machine, there are a number of commands that can be used for determining detailed, real-time machine hardware and configuration information.
• A table of some useful commands with example output is provided below. Hyperlinked commands display their man page

System Status Information

• LC Hardware page: hpc.llnl.gov/hardware
  • Unclassified systems only
• MyLC Portal: mylc.llnl.gov
  • Several portlets provide system status information:
    • machine status
    • login node status
    • scratch file system status
    • enclave status
  • Classified MyLC is at: lc.llnl.gov/lorenz/
• Machine status email lists:
  • Provide the most timely status information for system maintenance, problems, and system changes/updates
  • ocf-status and scf-status cover all machines on the OCF / SCF
  • Additionally, each machine has its own status list - for example: sierra-status@llnl.gov
• Login banner & news items - always displayed immediately after logging in
  • Login banner includes basic configuration information, announcements, and news items. Example login banner HERE.
  • News items (unread) appear at the bottom of the login banner. For usage, type news -h.
• Direct links for systems and file systems status pages:

Examples

Exercise 1

Logging in, basic configuration, and file systems information:

• Login to an LC cluster with X11 forwarding enabled
• Test X11
• Identify and SSH to other login nodes
• Familiarize yourself with the cluster's configuration
• Try the mxterm utility to access compute nodes
• Learn where/how to obtain hardware, OS and other configuration information for LC clusters
• Review basic file system info
• Try moving files to the HPSS storage system
• View file system status information

Software and Development Environment Overview

Development Environment Group (DEG)

• LC's Development Environment Group (DEG) provides a stable, usable, leading-edge parallel application development environment that enables users to improve the reliability and scalable performance of LLNL applications.
• DEG installs and supports the following LC software:
  • Compilers and Preprocessors
  • Debuggers
  • Memory Tools
  • Profiling Tools
  • Tracing Tools
  • Performance Analysis
  • Correctness Tools
  • Utilities
• Additionally, DEG's mission includes:
  • Working to make computing tools reliable, scalable and to help users make effective use of these tools.
  • Partnering with its application development user community to identify user requirements and evaluate tool effectiveness.
  • Collaborating with vendors and other third party software developers to ensure a complete environment in the most cost effective way possible and meet the needs of today's code developers utilizing emerging technologies.
• DEG Home Page: computing.llnl.gov/livermore-computing/development-environment-group

TOSS Operating System

• TOSS = Tri-Laboratory Operating System Stack
• Based on Red Hat Enterprise Linux (RHEL) with modifications to support targeted HPC hardware and cluster computing
• Used by most LC (and Tri-lab) production Linux clusters:
  • For Blue Gene systems, the login nodes use TOSS, but the compute nodes run a special Linux-like Compute Node Kernel (CNK).
  • CORAL EA and Sierra clusters use a "TOSS-like" OS/software stack, called blueos by LC.
• The primary components of TOSS include:
  • RHEL kernel optimized for large scale cluster computing
  • OpenFabrics Enterprise Distribution InfiniBand software stack including MVAPICH and OpenMPI libraries
  • Slurm workload manager
  • Integrated Lustre and Panasas parallel file system software
  • Scalable cluster administration tools
  • Cluster monitoring tools
  • GNU, C, C++ and Fortran90 compilers (GNU, Intel, PGI)
  • Testing software framework for hardware and operating system validation
• See Redhat's documentation for details on the RHEL kernel.
• Version information for LC's clusters:
  • TOSS: distro_version or cat /etc/toss-release
  • Redhat: cat /etc/redhat-release

Software Lists, Documentation, and Downloads

The table below lists and provides links to the majority of software available through LC and related organizations.

Modules

• Most LC clusters support Lmod modules for software packaging:
  • Provide a convenient, uniform way to select among multiple versions of software installed on LC systems.
  • Many LC software applications require that you load a particular "package" in order to use the software.
• Using Modules:

List available modules:     module avail
Load a module:              module add|load modulefile
Unload a module:            module rm|unload modulefile
List loaded modules:        module list
Read module help info:      module
Display module contents:    module display|show modulefile

• For more information see:
  • LC modules documentation
  • TACC documentation: LMOD: ENVIRONMENTAL MODULES SYSTEM
  • The module man page

LC Web Services - Confluence, JIRA, GitLab, etc.

• LC supports a suite of web-based collaboration tools:
  • Confluence: wiki tool, used for documentation, collaboration, knowledge sharing, file sharing, mockups, diagrams... anything you can put on a webpage.
  • JIRA: issue tracking and project management system
  • GitLab: Git repository hosting, Continuous Integration, Issue tracking
• All three collaboration tools:
  • Are based on LC usernames / groups and are intended to foster collaboration between LC users working on HPC projects.
  • Are installed on the CZ, RZ and SCF networks
  • Require authentication with your LC username and RSA PIN + token
  • Have a User Guide for usage information
• Locations:

Spack Package Manager

• Spack is a flexible package manager for HPC
• Easy to download and install. For example:

% git clone https://github.com/spack/spack
% . spack/share/spack/setup-env.csh (or setup-env.sh)

• There is an increasing number of software packages (over 4,200 as of May 2020) available for installation with Spack. Many open source contributions from the international community.
• To view available packages: spack list
• Then, to install a desired package: spack install packagename
• Additional Spack features:
  • Allows installations to be customized. Users can specify the version, build compiler, compile-time options, and cross-compile platform, all on the command line.
  • Allows dependencies of a particular installation to be customized extensively.
  • Non-destructive installs - Spack installs every unique package/dependency configuration into its own prefix, so new installs will not break existing ones.
  • Creation of packages is made easy.
• Extensive documentation is available at: spack.readthedocs.io

Compilers

Available Compilers and Invocation Commands

• The table below summarizes compiler availability and invocation commands on LC Linux clusters.
• For Sierra and CORAL EA compiler information, please see: hpc.llnl.gov/documentation/tutorials/using-lc-s-sierra-systems#compilers
• Note that parallel compiler commands are actually LC scripts that ultimately invoke the corresponding serial compiler.
• For details on the MPI parallel compiler commands, see hpc-tutorials.llnl.gov/mpi/

Compiler Versions and Defaults

• LC maintains multiple versions of each compiler.

• The Modules module avail command is used to list available compilers and versions:

    module avail intel

    module avail gcc

    module avail pgi

    module avail clang

• Versions: to determine the actual version you are using, issue the compiler invocation command with its "version" option. For example:

• To use an alternate version issue the Modules command: module load module-name

Compiler Options

• Each compiler has hundreds of options that determine what the compiler does and how it behaves.
• The options used by one compiler mostly differ from other compilers.
• Additionally, compilers have different default options.
• An in-depth discussion of compiler options is beyond the scope of this tutorial.
• See the compiler's documentation, man pages, and/or -help or --help option for details.

Compiler Documentation

• Intel and PGI: compiler docs are included in the /opt/compilername directory. Otherwise, see Intel or PGI web pages.
• GNU: see the web pages at gcc.gnu.org/
• LLVM/Clang: see the web pages at clang.llvm.org/docs/
• Man pages may/may not be available

Optimizations

• All compilers are able to perform optimizations, though they will differ between compilers even though the compiler flags appear to be the same.
• Optimizations are intended to make codes run faster, though this isn't guaranteed.
• Some optimizations "rewrite" your code, and can make debugging difficult, since the source may not match the executable.
• Optimizations can also produce wrong results, reduced precision, increased compile times and increased executable size.
• The table below summarizes common compiler optimization options. See the compiler documentation for details and other optimization options.

Floating-point Exceptions

• The IEEE floating point standard defines several exceptions (FPEs) that occur when the result of a floating point operation is unclear or undesirable:
  • overflow: an operation's result is too large to be represented as a float. Can be trapped, or else returned as a +/- infinity.
  • underflow: an operation's result is too small to be represented as a normalized float. Can be trapped, or else represented as as a denormalized float (zero exponent w/ non-zero fraction) or zero.
  • divide-by-zero: attempting to divide a float by zero. Can be trapped, or else returned as a +/- infinity.
  • inexact: result was rounded off. Can be trapped or returned as rounded result.
  • invalid: an operation's result is ill-defined, such as 0/0 or the sqrt of a negative number. Can be trapped or returned as NaN (not a number).
• By default, the Xeon processors used at LC mask/ignore FPEs. Programs that encounter FPEs will not terminate abnormally, but instead, will continue execution with the potential of producing wrong results.
• Compilers differ in their ability to handle FPEs. See the relevant compiler documentation for details.

Precision, Performance, and IEEE 754 Compliance

• Typically, most compilers do not guarantee IEEE 754 compliance for floating-point arithmetic unless it is explicitly specified by a compiler flag. This is because compiler optimizations are performed at the possible expense of precision.
• Unfortunately for most programs, adhering to IEEE floating-point arithmetic adversely affects performance.
• If you are not sure whether your application needs this, try compiling and running your program both with and without it to evaluate the effects on both performance and precision.
• See the relevant compiler documentation for details.

Mixing C and Fortran

• If you are linking C/C++ and FORTRAN code together, and need to explicitly specify the FORTRAN or C/C++ libraries on the link line.
• All of the other issues involved with mixed language programming apply, such as:
  • Column-major vs. row-major array ordering
  • Routine name differences - appended underscores
  • Arguments passed by reference versus by value
  • Common blocks vs. extern structs
  • Memory alignment differences
  • File I/O - Fortran unit numbers vs. C/C++ file pointers
  • C++ name mangling
  • Data type differences
• Auseful reference:
  • Using C/C++ and Fortran Together from www.yolinux.com/TUTORIALS/LinuxTutorialMixingFortranAndC.html

Debuggers

• Note This section only touches on selected highlights. For more information users will definitely need to consult the relevant documentation mentioned below. Also, please consult the "Supported Software and Computing Tools" web page located at hpc.llnl.gov/software.

TotalView

• TotalView is probably the most widely used debugger for parallel programs. It can be used with C/C++ and Fortran programs and supports all common forms of parallelism, including pthreads, openMP, MPI, accelerators and GPUs.
• Starting TotalView for serial codes: simply issue the command:

        totalview myprog

• Starting TotalView for interactive parallel jobs:
  • Some special command line options are required to run a parallel job through TotalView under SLURM. You need to run srun under TotalView, and then specify the -a flag followed by 1)srun options, 2)your program, and 3)your program flags (in that order). The general syntax is: totalview srun -a -n #processes -p pdebug myprog [prog args]
  • To debug an already running interactive parallel job, simply issue the totalview command and then attach to the srun process that started the job.
  • Debugging batch jobs is covered in LC's TotalView tutorial and in the "Debugging in Batch" section below.
• Documentation:
  • LC Tutorial
  • Vendor website: www.roguewave.com/

DDT

• DDT stands for "Distributed Debugging Tool", a product of Allinea Software Ltd.
• DDT is a comprehensive graphical debugger designed specifically for debugging complex parallel codes. It is supported on a variety of platforms for C/C++ and Fortran. It is able to be used to debug multi-process MPI programs, and multi-threaded programs, including OpenMP.
• Currently, LC has a limited number of fixed and floating licenses for OCF and SCF Linux machines.
• Usage information: see LC's DDT Quick Start information located at: hpc.llnl.gov/software/development-environment-software/allinea-ddt
• Documentation: see the vendor website: www.allinea.com

STAT - Stack Trace Analysis Tool

• The Stack Trace Analysis Tool gathers and merges stack traces from a parallel application's processes.
• STAT is particularly useful for debugging hung programs.
• It produces call graphs: 2D spatial and 3D spatial-temporal
  • The 2D spatial call prefix tree represents a single snapshot of the entire application (see image).
  • The 3D spatial-temporal call prefix tree represents a series of snapshots from the application taken over time.
• In these graphs, the nodes are labeled by function names. The directed edges, showing the calling sequence from caller to callee, are labeled by the set of tasks that follow that call path. Nodes that are visited by the same set of tasks are assigned the same color, giving a visual reference to the various equivalence classes.
• This tool should be in your default path as:
  • /usr/local/bin/stat-gui - GUI
  • /usr/local/bin/stat-cl - command line
  • /usr/local/bin/stat-view - viewer for DOT format output files
  • /usr/local/tools/stat - install directory, documentation
• More information: see the STAT web page at: hpc.llnl.gov/software/development-environment-software/stat-stack-trace-analysis-tool

Debugging in Batch: mxterm / sxterm

• Debugging batch parallel jobs on LC production clusters is fairly straightforward. The main idea is that you need to submit a batch job that gets your partition allocated and running.
• Once you have your partition, you can rsh to any of the nodes within it, and then starting running as though you’re in the interactive pdebug partition.
• For convenience, LC has developed the mxterm / sxterm utilities which makes the process even easier.
• How to use mxterm / sxterm:
  • If you are on a Windows PC, start your X11 application (such as X-Win32)
  • Make sure you enable X11 tunneling for your ssh session
  • ssh and login to your cluster
  • Issue the command as follows:
    

    mxterm #nodes #tasks #minutes
    sxterm #nodes #tasks #minutes

    
    
    Where: #nodes = number of nodes your job requires
    
    #tasks = number of tasks your job requires
    
    #minutes = how long you need to keep your partition for debugging

  • This will submit a batch job for you that will open an xterm when it starts to run.
  • After the xterm appears, cd to the directory with your source code and begin your debug session.
  • This utility does not have a man page, however you can view the usage information by simple typing the name of the command.

Other Debuggers

• Several other common debuggers are available on LC Linux clusters, though they are not recommended for parallel programs when compared to TotalView and DDT.
• PGDBG: the Portland Group Compiler debugger. Documentation: www.pgroup.com/products/pgdbg.htm
• GDB: GNU GDB debugger, a command-line, text-based, single process debugger. Documentation: www.gnu.org/software/gdb
• DDD: GNU DDD debugger is a graphical front-end for command-line debuggers such as GDB, DBX, WDB, Ladebug, JDB, XDB, the Perl debugger, the bash debugger, or the Python debugger. Documentation: www.gnu.org/software/ddd

A Few Additional Useful Debugging Hints

• Core Files:
  • It is quite likely that your shell's core file size setting may limit the size of a core file so that it is inadequate for debugging, especially with TotalView.
  • To check your shell's limit settings, use either the limit (csh/tcsh) or ulimit -a (sh/ksh/bash) command. For example:

        limit
        cputime      unlimited
        filesize     unlimited
        datasize     unlimited
        stacksize    unlimited
        coredumpsize 16 kbytes
        memoryuse    unlimited
        vmemoryuse   unlimited
        descriptors  1024
        memorylocked 7168 kbytes
        maxproc      1024

        ulimit -a
        address space limit (kbytes)   (-M)  unlimited
        core file size (blocks)        (-c)  32
        cpu time (seconds)             (-t)  unlimited
        data size (kbytes)             (-d)  unlimited
        file size (blocks)             (-f)  unlimited
        locks                          (-L)  unlimited
        locked address space (kbytes)  (-l)  7168
        nofile                         (-n)  1024
        nproc                          (-u)  1024
        pipe buffer size (bytes)       (-p)  4096
        resident set size (kbytes)     (-m)  unlimited
        socket buffer size (bytes)     (-b)  4096
        stack size (kbytes)            (-s)  unlimited
        threads                        (-T)  not supported
        process size (kbytes)          (-v)  unlimited

• To override your default core file size setting, use one of the following commands:

• Some users have complained that for many-process jobs, they actually don't want core files or only want small core files because normal core files can fill up their disk space. The limit (csh/tcsh) or ulimit -c (sh/ksh/bash) commands can be used as shown above to set smaller / zero sizes.
• Add the sinfo and squeue commands to your batch scripts to assist in diagnosing problems. In particular, these commands will document which nodes your job is using.
• Also add the -l option to your srun command so that output statements are prepended with the task number that created them.
• Be sure to check the exit status of all I/O operations when reading or writing files in Lustre. This will allow you to detect any I/O problems with the underlying OST servers.
• If you know/suspect that there are problems with particular nodes, you can use the srun -x option to skip these nodes. For example: srun -N12 -x "cab40 cab41" -ppdebug myjob

Performance Analysis Tools

We Need a Book!

• The subject of Performance Analysis Tools is far too broad and deep to cover here. Instead, a few pointers are being provided for those who are interested in further research.
• The first place to check are the "Development Environment Software" web pages at: hpc.llnl.gov/software/development-environment-software for what may be available here. Some example tools are listed below.

Memory Correctness Tools

Memcheck: Valgrind's Memcheck tool detects a comprehensive set of memory errors, including reads and writes of unallocated or freed memory and memory leaks.

TotalView: Allows you to stop execution when heap API problems occur, list memory leaks, paint allocated and deallocated blocks, identify dangling pointers, hold onto deallocated memory, graphically browse the heap, identify the source line and stack backtrace of an allocation or deallocation, summarize heap use by routine, filter and dump heap information, and review memory usage by process or by library.

memP: The memP tool provides heap profiling through the generation of two reports: a summary of the heap high-water-mark across all processes in a parallel job as well as a detailed task-specific report that provides a snapshot of the heap memory currently in use, including the amount allocated at specific call sites.

Intel Inspector: Primarily a thread correctness tool, but memory debugging features are included.

Profiling, Tracing, and Performance Analysis

Open|SpeedShop: Open|SpeedShop is a comprehensive performance tool set with a unified look and feel that covers most important performance analysis steps. It offers various different interfaces, including a flexible GUI, a scripting interface, and a Python class. Supported experiments include profiling using PC sampling, inclusive and exclusive execution time analysis, hardware counter support, as well as MPI, I/O, and floating point exception tracing. All analysis is applied on unmodified binaries and can be used on codes with MPI and/or thread parallelism.

TAU: TAU is a robust profiling and tracing tool from the University of Oregon that includes support for MPI and OpenMP. TAU provides an instrumentation API, but source code can also be automatically instrumented and there is support for dynamic instrumentation as well. TAU is generally viewed as having a steep learning curve, but experienced users have applying the tool with good results at LLNL. TAU can be configured with many feature combinations. If the features you are interested in are not available in the public installation, please request the appropriate configuration through the hotline. TAU developer response is excellent, so if you are encountering a problem with TAU, there is a good chance it can be quickly addressed.

HPCToolkit: HPCToolkit is an integrated suite of tools for measurement and analysis of program performance on computers ranging from multicore desktop systems to the largest supercomputers. It uses low overhead statistical sampling of timers and hardware performance counters to collect accurate measurements of a program's work, resource consumption, and inefficiency and attributes them to the full calling context in which they occur. HPCToolkit works with C/C++/Fortran applications that are either statically or dynamically linked. It supports measurement and analysis of serial codes, threaded codes (pthreads, OpenMP), MPI, and hybrid (MPI + threads) parallel codes.

mpiP: A lightweight MPI profiling library that provides time spent in MPI functions by callsite and stacktrace. This tool is developed and maintained at LLNL, so support and modifications can be quickly addressed. New run-time functionality can be used to generate mpiP data without relinking through the srun-mpip and poe-mpip scripts on Linux and AIX systems.

gprof: Displays call graph profile data. The gprof command is useful in identifying how a program consumes CPU resources. Gprof does simple function profiling and requires that the code be built and linked with -pg. For parallel programs, in order to get a unique output file for each process, you will need to set the undocumented environment variable GMON_OUT_PREFIX to some non-null string. For example: setenv GMON_OUT_PREFIX 'gmon.out.'`/bin/uname -n`

pgprof: PGI profiler - pgprof is a tool which analyzes data generated during execution of specially compiled programs. This information can be used to identify which portions of a program will benefit the most from performance tuning.

PAPI: Portable hardware performance counter library.

PapiEx: A PAPI-based performance profiler that measures hardware performance events of an application without having to instrument the application.

VTune Amplifier: The Intel VTune Amplifier tool is a performance analysis tool for finding hotspots in serial and multithreaded codes. Note the installation on LC machines does not include the advanced hardware analysis capabilities.

Intel Profiler: The Intel Profiler tool is built into the Intel compiler along with a simple GUI to display the collected results.

Vampir / Vampirtrace: Full featured trace file visualizer and library for generating trace files for parallel programs.

Beyond LC

• Beyond LC, the web offers endless opportunities for discovering tools that aren't available here.
• In many cases, users can install tools in their own directories if LC doesn't have/support them.

Graphics Software and Resources

• LC's Information Management and Graphics Group (IMGG) supports a wide range of graphics related software packages.
• For a complete list and getting started instructions see: hpc.llnl.gov/software/visualization-software

Consulting

• Consulting on scientific visualization issues (demos, classes, getting started, visualization advice)
• Maintaining a graphics environment that is current and consistent across LC computing platforms
• Graphics software support
• Troubleshooting graphics related problems
• Also available to work with customers to develop custom data visualization applications

Video Production

• State-of-the-art unclassified and classified facilities
• 3D modeling and computer animation
• Video and audio editing; quick turnaround of videos
• Also, photo quality output and DVD/CDROM creation

Visualization Machine Resources

• LC provides dedicated clusters for visualization work.
• Accounts must be requested through the LC Hotline for use of the visualization machines.

Power Walls

Large screen displays for viewing high-resolution images and data comparisons

Running Jobs

Where to Run?

Note This section only provides a general overview on running jobs on LC systems. Details associated with running jobs are covered in depth in other LC tutorials (Slurm and Moab, MPI, BG/Q, OpenMP, Pthreads, etc.)

Determining Your Job's Requirements

• The first thing that needs to be done before deciding where to run your jobs is to determine your jobs' requirements.
• There are several important factors to consider, because not all LC clusters may match your jobs' requirements:

Getting Machine Configuration Information

• After determining your jobs' requirements, the next step is to see which LC machines match those requirements.
• LC's production clusters vary widely in their configurations.
• Fortunately, getting configuration details for any production LC machine is easy to do.
• See the previous System Status and Configuration Information section for details.

Job Limits

• For all production clusters, there are defined job limits which vary from cluster to cluster. The primary job limits apply to:
  • How many nodes/cores a job may use
  • How long a job may run
  • How many simultaneous jobs can a user run?
  • What time of the day/week can jobs run?
• Most job limits are enforced by the batch system
• Some job limits are enforced by a "good neighbor" policy
• An easy way to determine the job limits for a machine where you are logged in is to use the command: news job.lim.machinename where machinename is the name of the machine you are logged into.
• Job limits are also documented on the "MyLC" web pages:
  
  OCF-CZ: mylc.llnl.gov
  
  OCF-RZ: rzmylc.llnl.gov
  
  SCF: lc.llnl.gov/lorenz
  
  Just click on any machine name in the "machine status" or "my accounts" portlets. Then select the "job limits" tab.
• Further discussion, and a summary table of job limits for all production machines are available in the Queue Limits section of the Slurm tutorial.

Accounts and Banks

• In order to run on a machine, you need a valid login account, and at least one "bank" to charge your usage against.
• Getting information about your assigned bank(s) is covered in the Banks and Banks and Usage Information sections of the Slurm and Moab tutorial.

Serial vs Parallel

• Parallel jobs can range in size from a single node (multi-core) to the full system.
• All but a few of LC's production clusters are intended to be used for parallel jobs.
• Serial jobs by definition require only one core on a node.
• Because running serial jobs on parallel clusters would waste compute resources, LC provides several clusters where serial jobs can be run without wasting compute resources.
• Running serials jobs is discussed here: hpc.llnl.gov/banks-jobs/running-jobs/slurm-and-moab#Serial

Dedicated Application Time (DAT)

• Some LC systems allow users to request dedicated machine use.
• DAT is primarily for "big runs" that require a larger number of nodes and/or longer time than the normal batch limits permit.
• During DATs, other user jobs are drained/removed and the batch queue is suspended while the dedicated job runs.
• DATs are posted to the relevant machine status news list.
• DATs are requested through the LC Hotline. Forms for requesting "Expedited Priority Runs" are available at: hpc.llnl.gov/accounts/forms.

Batch Versus Interactive

Interactive Jobs

• Refers to jobs that you launch and interact with real-time from the command line.
• Don't use Login Nodes (in most cases):
  • Login nodes are shared, oftentimes by many users.
  • Intended for non-cpu intensive interactive tasks, such as editing, working with files, running GUIs, browsers, debuggers, etc.
  • Not intended for running CPU-intensive production jobs, which can negatively impact other users.
• Most clusters have a pdebug partition of compute nodes dedicated to small, short, interactive jobs. Another common interactive partition is pvis.
• There are several ways to run interactive jobs on LC's Linux clusters, as described below.
  • For interactive jobs on CORAL clusters, see: hpc.llnl.gov/documentation/tutorials/using-lc-s-sierra-systems#Interactive.
• Method 1: Use srun on a login node specifying compute nodes in the pdebug or other interactive partition.
  • To view the partitions on a cluster use the sinfo -s command. It shows which partitions are configured, their time limit and how the nodes are currently being used (Allocated / Idle / Other / Total). For example, on Quartz note the pdebug partition:
    

    % sinfo -s
    PARTITION AVAIL  TIMELIMIT   NODES(A/I/O/T)  NODELIST
    pdebug       up      30:00       20/26/0/46  quartz[1-46]
    pbatch*      up 1-00:00:00  2865/52/13/2930  quartz[47-186,193-378,...,2887-3072]
    pall       down   infinite  2885/78/13/2976  quartz[1-186,193-378,...,2887-3072]
    

  • Then use the srun command to run the job on compute nodes in pdebug. For example, to run your executable on 2 pdebug nodes using 16 tasks:
    

    % srun -p pdebug -N 2 -n 16 myexe

• Method 2: Use salloc to acquire compute nodes for interactive use.
  • On a login node, request an allocation of compute nodes, and optionally the partition to acquire them from. The default is usually pbatch, but small short jobs usually get better turnaround time in pdebug. For example, to request 2 pdebug compute nodes:
    

    % salloc -p pdebug -N 2
    salloc: Pending job allocation 3891090
    salloc: job 3891090 queued and waiting for resources
    salloc: job 3891090 has been allocated resources
    salloc: Granted job allocation 3891090

  • After your allocation is granted, you will be placed on the first node of the allocation. You can verify this as follows:
    

    % squeue -u joeuser
      JOBID PARTITION     NAME      USER ST       TIME  NODES NODELIST(REASON)
    3891090    pdebug       sh   joeuser  R       0:07      2 quartz[1-2]
    % hostname
    quartz1

  • You can now run whatever you'd like interactively on your job's compute nodes.
• Method 3: Use sxterm / mxterm to acquire compute nodes for interactive use.
  • Similar to salloc, but different because it will actually "behind the scenes" create a batch job for your request, queue it and then when it starts to run on a compute node, pop open an xterm window on your desktop.
  • For example requesting 1 node / 1 task / for 30 minutes in the pdebug partition:
    

    % sxterm 1 1 30 -p pdebug
    Submitted batch job 3048379

  • Wait until the xterm window appears on your desktop. You will be on the first node of your job's allocation and can run whatever you'd like interactively.
  • There's no man page - just invoke "sxterm" with no options and a usage message will display.
  • The mxterm command is similar, with different options. Just invoke "mxterm" for usage information.
• Method 4: Submit a batch job and then rsh/ssh to the compute nodes where it is running.
  • You can submit a batch job (covered next) that really doesn't do anything except "sleep".
  • The squeue command can be used to show when and where your job is running.
  • After it starts running, you can rsh or ssh to any of its compute nodes.
  • Once on a compute node, you can run whatever you'd like interactively.
• Important usage notes about the pdebug partition:
  • As the name pdebug implies, interactive jobs should be short, small debugging jobs, not production runs.
  • Shorter time limit
  • Fewer number of nodes permitted
  • There is usually a "good neighbor" policy in effect - don't monopolize the queue or setup streams of jobs

Batch Jobs

Note This section only provides a quick summary of batch usage on LC's Linux clusters. For details, see the Slurm and Moab tutorial. For CORAL/Sierra systems see the Sierra tutorial.

• Typically, most of a cluster's compute nodes are configured into a pbatch partition.
• The pbatch partition is intended for production work:
  • Longer time limits
  • Larger number of nodes per job
  • Limits enforced by batch system rather than "good neighbor" policy
• The pbatch partition is managed by the workload manager
• Batch jobs must be submitted in the form of a job control script. Example batch job script:

#!/bin/tcsh
##### These lines are for Slurm
#SBATCH -N 16
#SBATCH -J parSolve34
#SBATCH -t 2:00:00
#SBATCH -p pbatch
#SBATCH --mail-type=ALL
#SBATCH -A myAccount
#SBATCH -o /p/lustre1/joeuser/par_solve/myjob.out

##### These are shell commands
date
cd /p/lustre1/joeuser/par_solve
##### Launch parallel job using srun - assumes 36 cores/node
srun -n576 a.out
echo 'Done'

• Example batch job script submission commands:

sbatch myjobscript
sbatch myjobscript -p ppdebug -A mic
sbatch myjobscript -t 45:00

• After successfully submitting a job, you may then check its progress and interact with it (hold, release, alter, terminate) by means of other batch commands - discussed in the Interacting with Jobs section.
• Some clusters have additional partitions permitting batch jobs.
• Interactive use of pbatch nodes is facilitated by using the mxterm command - discussed in the Where to Login section.
• Interactive debugging of batch jobs is possible - covered in the Debuggers section.

Starting Jobs - srun

Note This section only provides a quick summary of batch usage on LC's Linux clusters. For details, see the Slurm and Moab tutorial.

The srun Command

• The SLURM srun command is required to launch parallel jobs - both batch and interactive.
• It should also be used to launch serial jobs in the pdebug and other interactive queues.
• Syntax:
  
  
  
  srun   [option list]   [executable]   [args]
  
  
  
  Note that srun options must precede your executable.

• Interactive use example, from the login node command line. Specifies 2 nodes (-N), 16 tasks (-n) and the interactive pdebug partition (-p):

% srun -N2 -n16 -ppdebug myexe

• Batch use example requesting 16 nodes and 256 tasks (assumes nodes have 16 cores):
  • First create a job script that requests nodes via #SBATCH -N and uses srun to specify the number of tasks and launch the job.

#!/bin/csh
#SBATCH -N 16
#SBATCH -t 2:00:00
#SBATCH -p pbatch
# Run info and srun job launch
cd /p/lustre2/joeuser/par_solve
srun -n256 a.out
echo 'Done'

• Then submit the job script from the login node command line:

% sbatch myjobscript

• Primary differences between batch and interactive usage:

• More Examples:

• Behavior of srun -N and -n flags - using 4 nodes in batch, each of which has 16 cores:

srun Options

• srun is a powerful command with @100 options affecting a wide range of job parameters.
• For example:
  • Accounting
  • Number and placement of processes/threads
  • Process/thread binding
  • Job resource requirements; dependencies
  • Mail notification options
  • Input, output options
  • Time limits
  • Checkpoint, restart options
  • and much more....
• Some srun options may be set via @60 SLURM environment variables. For example, SLURM_NNODES behaves like the -N option.
• See the srun man page for details.

Interacting with Jobs

Note This section only provides a quick summary of batch usage on LC's Linux clusters. For details, see the Slurm and Moab tutorial. For CORAL/Sierra systems see the Sierra tutorial.

Monitoring Jobs and Displaying Job Information

• There are several different job monitoring commands. Some are based on Moab, some on Slurm, and some on other sources.
• The more commonly used job monitoring commands are summarized in the table below with links to additional information and examples.

Holding / Releasing Jobs

• Jobs can be placed "on hold" when they are submitted.
• They can also be placed on hold while they are waiting to run.
• Held jobs can then be released to run later
• More information/examples: see Holding and Releasing Jobs in the Slurm and Moab tutorial.

Modifying Jobs

• After a job has been submitted, certain attributes may be modified using the scontrol update and mjobctl -m commands.
• Examples of parameters that can be changed: account, queue, job name, wall clock limit...
• More information/examples: see Changing Job Parameters in the Slurm and Moab tutorial.

Terminating / Canceling Jobs

• Interactive srun jobs launched from the command line should normally be terminated with a SIGINT (CTRL-C):
  • The first CTRL-C will report the state of the tasks
  • A second CTRL-C within one second will terminate the tasks
• For batch jobs, the mjobctl -c and canceljob commands can be used.
• More information/examples: see Canceling Jobs in the Slurm and Moab tutorial.

Other Topics of Interest

Optimizing Core Usage

• Fully utilizing the cores on a node requires that you use the right combination of srun and Slurm/Moab options, depending upon what you want to do and which type of machine you are using.
• MPI only: for example, if you are running on a cluster that has 36 cores per node, and you want your job to use all 36 cores on 4 nodes (36 MPI tasks per node), then you would do something like:

• MPI with Threads: If your MPI job uses POSIX or OpenMP threads within each node, you will need to calculate how many cores will be required in addition to the number of tasks. For example, running on a cluster having 16 cores per node, an 8-task job where each task creates 4 OpenMP threads, would need a total of 32 cores, or 2 nodes:

8 tasks * 4 threads / 16 cores/node = 2 nodes

• You can include multiple srun commands within your batch job command script. For example, suppose that you were conducting a scalability run on a 36 core/node Linux cluster. You could allocate the maximum number of nodes that you would use with #SBATCH -N and then have a series of srun commands that use varying numbers of nodes:

#SBATCH -N 8

srun -N1 -n36 myjob
srun -N2 -n72 myjob
srun -N3 -n108 myjob
....
srun -N8 -n228 myjob

Diskless Nodes

• Because LC's Linux clusters employ a 64-bit architecture, 16 exabytes of memory can be addressed - which is about 4 billion times more than 4 GB limit of 32-bit architectures. By current standards, this is virtually unlimited memory.
• In reality, machines are usually configured with only GBs of memory, so any address access that exceeds physical memory will result (on most systems) with paging and degraded performance.
• However, LC machines are diskless and have no paging space.
• This has very important implications for programs that exceed physical memory. For example, most compute nodes have only 16-64 GB of physical memory.
• Programs that exceed physical memory will terminate with an OOM (out of memory) error and/or segmentation fault.

Process/Thread Binding to Cores

• By default, jobs run on LC's Linux clusters have their processes bound to the available cores on a node. For example:

% srun -l -n2 numactl -s | grep physc | sort
0: physcpubind: 0 1 2 3 4 5 6 7
1: physcpubind: 8 9 10 11 12 13 14 15

% srun -l -n4 numactl -s | grep physc | sort
0: physcpubind: 0 1 2 3
1: physcpubind: 4 5 6 7
2: physcpubind: 8 9 10 11
3: physcpubind: 12 13 14 15

% srun -l -n8 numactl -s | grep physc |sort
0: physcpubind: 0 1
1: physcpubind: 2 3
2: physcpubind: 4 5
3: physcpubind: 6 7
4: physcpubind: 8 9
5: physcpubind: 10 11
6: physcpubind: 12 13
7: physcpubind: 14 15

• Binding processes to cores may improve performance by keeping cache local to cores.
• If a process is multi-threaded (such as with OpenMP), the threads will run on any of the cores bound to their process.
• To bind an OpenMP thread to a single core, the OMP_PROC_BIND environment variable can be used (set to "TRUE").
• Additionally, LC provides a couple useful utilities for binding processes and threads to cores:
  • mpibind - automatically binds processes and threads to cores. Documentation at lc.llnl.gov/confluence/display/TLCC2/mpibind (requires authentication)
  • mpifind - reports on how processes and threads are bound to cores. Documentation at lc.llnl.gov/confluence/display/TLCC2/mpifind (requires authentication)

Vectorization

• Historically, the Xeon architecture has provided support for SIMD (Single Instruction Multiple Data) vector instructions through Intel's Streaming SIMD Extensions (SSE, SSE2, SSE3, SSE4) instruction sets.
• AVX - Advanced Vector Extension instruction set (2011) improved on SSE instructions by increasing the size of the vector registers from 128-bits to 256-bits. AVX2 (2013) offered further improvements, such as fused multiply-add (FMA) instructions to double FLOPS.
• The primary purpose of these instructions is to increase CPU throughput by performing operations on vectors of data elements, rather than on single data elements. For example:

• Sandy Bridge-EP (TLCC2) processors support SSE and AVX instructions.
• Broadwell (CTS-1) processors support SSE, AVX and AVX2.
• To take advantage of the potential performance improvements offered by vectorization, all you need to do is compile with the appropriate compiler flags. Some recommendations are shown in the table below.

• For additional information see the Vectorization section of the Linux Clusters tutorial.

Hyper-threading

• On Intel processors, hyper-threading enables 2 hardware threads per core.
• Hyper-threading benefits some codes more than others. Tests performed on some LC codes (pF3D, IMC, Ares) showed improvements in the 10-30% range. Your mileage may vary.
• On TOSS 3 systems, hyper-threading is turned on by default. Details are available at: lc.llnl.gov/confluence/display/TCE/Hyper-Threading (authentication required).

Clusters Without an Interconnect (serial and single-node jobs)

• The agate, borax and rztrona clusters fall into this category.
• Limited to serial and single-node parallel jobs.
• Multiple users and jobs can run on a single node (shared node).
• Jobs are allocated ONE core by default. For this reason, it is very important that you tell the scheduler how many cores your job actually requires:
  • For batch jobs, be sure to include in your job script:
    • Slurm: #SBATCH -n #cores
    • Moab: #MSUB -l ttc=#cores
  • For interactive jobs, be sure to use the srun -c flag.
• MPI jobs: these are started with the srun command as described previously, but because there is no interconnect, jobs are limited to a single node and communications are done in shared memory.
• Pthreads and OpenMP jobs can be run as usual - keeping in mind the required number of cores per node, and that a node may be shared with other users.
• Important: Please see the Running on Serial Clusters section of the Moab and SLURM tutorial for further discussion on running jobs on these clusters.

Batch Systems

• The batch system used depends upon the cluster:
  • Linux TOSS 3 clusters: Run Slurm stand-alone with Moab wrappers
  • CORAL Early Access clusters: Run Spectrum LSF with no wrappers
• Because there is so much to discuss, we won't say much more here.
• The Slurm and Moab batch systems are covered in-depth in LC's Slurm and Moab tutorial and Running Jobs.
• Spectrum LSF is covered under Running Jobs.

Miscellaneous Topics

Clusters with GPUs

• Several LC Linux clusters have GPUs:
  • Max (SCF)
  • Pascal, Surface (CZ)
  • RZHasgpu (RZ)
• Additionally, the Sierra and CORAL Early Access clusters have GPUs:
  • Sierra, Shark (SCF)
  • Lassen, Ray (CZ)
  • RZAnsel, RZManta (RZ)
• These are discussed on the respective webpages:
  • Linux clusters: hpc.llnl.gov/documentation/tutorials/livermore-computing-linux-commodity-clusters-overview-part-two#GPU
  • Sierra and CORAL EA clusters: hpc.llnl.gov/documentation/tutorials/using-lc-s-sierra-systems

Big Data at LC

• Recently "Big Data" cluster computing frameworks have become popular. There has been interest in running some frameworks, such as Hadoop and Spark, on LC clusters.
• LC clusters differ from most "Big Data" cluster architectures:
  • LC uses a different scheduler/resource manager (Moab/Slurm)
  • LC uses a networked file system (Lustre)
  • Most LC systems have no local disk
• LC provides Magpie, a set of scripts to provide a framework to run several popular "Big Data" frameworks on LC systems.
  • Setup a Hadoop, Spark, etc. environment to run Big Data jobs
  • Work with LC file systems and scheduler/resource manager
  • Hide a number of LC-isms from users
  • Provide secure mechanisms to run within LC environment
  • Magpie instructions and information can be found on LC Confluence: lc.llnl.gov/confluence/display/BigData/Magpie+Guide
• Catalyst is a special LC cluster dedicated to Big Data research and testing.
  • 324 node Intel Xeon E5-2695 v2 nodes; compute nodes have 24 cores/node
  • Compute nodes have 800 NVRAM SSD storage
  • Catalyst information available at: lc.llnl.gov/confluence/display/BigData/Catalyst+Information+and+Documentation

Green Data Oasis (GDO)

• Green Data Oasis (GDO) is a large data store (1.4 PB as of Nov 2019) on the LLNL open network.
• Purpose:
  • Facilitate the sharing of scientific data with external collaborators by providing an easy way to share data
  • Designed as a data portal and is not intended for doing data analysis or other CPU-intensive activities.
• LC managed program for the Multiprogrammatic and Institutional Computing (M&IC) Initiative and Laboratory Science and Technology Office (LSTO).
• More information: hpc.llnl.gov/services/green-data-oasis

Security Reminders

• Follow procedures when escorting into limited areas
• Restrictions on cell phones, cameras, lap-tops and other items
• Separation between classified and unclassified equipment, including media
• Never share your password/account with anyone (duh!!). SCF passwords should be regarded as Secret Restricted Data (SRD)
• Keep classified information off unclassified systems - including email
• Ask an Authorized Derivative Classifier (ADC) if you're not sure
• LC can assist with the disposal of classified data
• For the full story, visit the LLNL security web page located at: security-r.llnl.gov (LLNL internal)

Where to Get Information & Help

LC Hotline

• The LC Hotline staff provide walk-in, phone and email assistance weekdays 8:00am - noon, 1:00pm - 4:45pm.
• Walk-in Consulting:
  • On-site users can visit the LC help desk consultants in Building 453, Room 1103.
  • Note that this is a Q-clearance area.
  • Need a map?
• Phone:
  • (925) 422-4531 - Main number
  • 422-4532 - Direct phone line for technical consulting help
  • 422-4533 - Direct phone line for support help (accounts, passwords, forms, etc)
• Email:
  • Technical Help: lc-hotline@llnl.gov
  • Accounts & Passwords: lc-support@llnl.gov

LC Users Home Page: hpc.llnl.gov

• hpc.llnl.gov: LC maintains extensive web documentation for all systems and also for computing in the LC environment:
• A few highlights:
  • Accounts - how to request an account; forms
  • Access Information - how to access and login to LC systems
  • Training - online tutorials and workshops
  • Compute Platforms - complete list with details for all LC systems
  • Machine Status - shows current OCF machines status with links to detailed information such as MOTD, currently running jobs, configuration, announcements, etc.
  • Software - including compilers, tools, debuggers, visualization, math libs
  • Running Jobs - including using LC's workload managers
  • Documentation - user manuals for a range of topics, technical bulletins, user meetings slides
  • Getting Help - how to contact the LC Hotline
• Some web pages are password protected. If prompted to enter a userid/password, use your OTP login.
• Some web pages may only be accessed from LLNL machines or by using one of the LC Remote Access Services covered previously.

Lorenz User Dashboard: mylc.llnl.gov

• Provides a wealth of real-time information in a user-friendly dashboard
• Simply enter "mylc" into your browser's address bar. The actual URL is: lc.llnl.gov/lorenz/mylc/mylc.cgi
• Click on the screenshot at right to see a larger version. Note: If your browser squeezes the large image into a single window, try zooming to get more detail.

Login Banner

• Login banner / MOTD may be very important!
  • News topics for LC, for the login system
  • Some configuration information
  • Useful references and contact information
  • System status information
  • Quota and password expiration warnings also appear when you login

News Items

• News postings on each LC System:
  • Unread news items appear with login messages
  • news -l - list all news items
  • news -a - display content of all news messages
  • news -n - lists unread messages
  • news -s - shows number of unread items
  • news item - shows specified news item
  • You can also list/read the files in /var/news on any system. This is useful when your searching for a topic you've already read and can't remember the news item name. You can also "grep" on these files.

• Also accessible from hpc.llnl.gov and Lorenz.

Machine Email Lists

• Machine status email lists exist for all LC machines
• Provide important, timely information not necessarily announced elsewhere
• ocf-status@llnl.gov and scf-status@llnl.gov are general lists for all users
• Plus each machine has its own list, for example: zin-status@llnl.gov.
• The LC Hotline initially populates a list with subscribers, but you can subscribe/unsubscribe yourself anytime using the listserv.llnl.gov website.

LC User Meetings

• When held, is usually scheduled for the first Tuesday of the month at 9:30 am
• Building 132 Auditorium (or as otherwise announced)
• Agenda and viewgraphs on LC Home Page (hpc.llnl.gov) See "Documentation" and look for "User Meeting Viewgraphs". Note that these are LLNL internal web pages.

Exercise 2

Compiling, running, job and system status information:

• Get information about running and queued jobs
• Get compiler information
• Compile and run serial programs
• Compile and run parallel MPI and OpenMP programs, both interactively and in batch
• Check hyper-threading
• Get online system status information (and more)

Glossary of Terminology

 Facilities, Machine Room Tours, Photos

Facilities

 

• Most of LC's computing resources are located in the Livermore Computing Complex (LCC) building 453, and buildings 451 and 654. The LCC was formerly known as the Terascale Simulation Facility (TSF).
• Map available here

• LCC highlights:
  • Four-story office tower with 121,600 square feet for 285 offices, a visualization theater, a 150-seat auditorium, and several conference rooms on each floor.
  • Machine room with 48,000 square feet of unobstructed computer room floor
  • 30 megawatts machine power capacity
  • Mechanical cooling system with cooling towers boasting total capacity of 12,600 gallons per minute, a chiller plant with total capacity of 7,200 tons, and air handlers with a total capacity of 2,720,000 cubic feet per minute
  • 3,600-gallon-per-minute, closed-loop, liquid-cooling system for Sequoia that can cool up to 9.6 megawatts.
• LC's building 654 comprises 6,000 square feet of computer floor space and is scalable up to 7.5 MW. B654 schematic drawing
• Additional reading/viewing:
  • asc.llnl.gov/facilities

Machine Room Tours

• LLNL hosts can request tours of the B453 machine room for visitors and groups. Hosts are responsible for providing Administrative Escorts (AE) and ensuring AE policies/rules are followed.
• Tour participants must be U.S. citizens.
• To request a tour, LLNL staff please see directions on SD Tour Information. Click the "Request Tours" button to submit a form.
• Summer students: See the Lab Events Calendar for details and registration: https://events.llnl.gov/.

Machine Photos

• Photo collections of some LC systems, present and past, are available at this internal URL that requires authentication: lc.llnl.gov/confluence/display/gallery/Photo+Gallery+of+LC+Systems