Sierra 

Sierra, Livermore’s latest advanced technology high performance computing system, joined LLNL’s lineup of supercomputers in 2018. The new system provides computational resources that are essential for nuclear weapon scientists to fulfill the National Nuclear Security Administration’s stockpile stewardship mission through simulation in lieu of underground testing. Advanced Simulation and Computing (ASC) Program scientists and engineers will use Sierra to assess the performance of nuclear weapon systems as well as nuclear weapon science and engineering calculations. These calculations are necessary to understand key issues of physics, the knowledge of which later makes its way into the integrated design codes. This work on Sierra has important implications for other global and national challenges such as nonproliferation and counterterrorism.

The IBM-built Sierra supercomputer is providing over six times the sustained throughput performance and over five times the sustained scalable science performance of Sequoia, with a 125 petaflops peak. Sierra, which combines two types of processor chips — IBM’s Power 9 processors and NVIDIA’s Volta graphics processing units, is over five times more power efficient than Sequoia, with a peak power consumption of approximately 11 megawatts. 

Additional Resources

For the Public

For Users

*Login nodes: sierra[4358-4362]

NOTE: most numbers are for compute nodes only - not login or service nodes.

**NOTE: System memory on Sierra is more accurately reflected as: 1.29 PiB

Job Limits

Each LC platform is a shared resource. Users are expected to adhere to the following usage policies to ensure that the resources can be effectively and productively used by everyone. You can view the policies on a system itself by running:

news job.lim.MACHINENAME

Web Version of Sierra Job Limits

There are 4320 compute nodes with 40 POWER9 cores, 4 NVIDIA Volta V100 GPUs and 256 GB of memory on each node.

Jobs are scheduled per node. Sierra has two main scheduling pools (queues):

  • pdebug—36 nodes
  • pbatch—4284 nodes
Queue          Max nodes / job    Max runtime
---------------------------------------------
pdebug            18(*)             2 hrs
pbatch            2048               24 hrs
---------------------------------------------

(*) pdebug is intended for debugging, visualization, and other inherently interactive work. It is NOT intended for production work. Do not use pdebug to run batch jobs. Do not chain jobs to run one after the other. Do not use more than half of the nodes during normal business hours. Individuals who misuse the pdebug queue in this or any similar manner will be denied access to running jobs in the pdebug queue.

Hardware

Each node has two 22-core 3.45 GHz IBM POWER9 processors. Two of the cores on each socket are reserved for system use, leaving 40 usable cores per node. The vast majority of the cycles on each node are provided by four NVIDIA Volta V100 GPUs per node. Each node also has 256 GB of system memory and 64 GB of GPU memory. The nodes are connected by Mellanox EDR InfiniBand.

Documentation

Please call or send email to the LC Hotline if you have questions. | LC Hotline:  phone: 925-422-4531 | SCF email: lc-hotline@pop.llnl.gov
Zone
SCF
Vendor
IBM
User-Available Nodes
Login Nodes*
5
Batch Nodes
4,284
Debug Nodes
36
Total Nodes
4,474
CPUs
CPU Architecture
IBM Power9
Cores/Node
44
Total Cores
190,080
GPUs
GPU Architecture
NVIDIA V100 (Volta)
Total GPUs
17,280
GPUs per compute node
4
Memory Total (GiB)
1,382,400
CPU Memory/Node (GiB)
256
Peak Performance
Peak PFLOPS (CPUs)
4.666
Peak PFLOPs (GPUs)
120.960
Peak PFLOPS (CPUs+GPUs)
125.626
Clock Speed (GHz)
3.4
Peak single CPU memory bandwidth ((GiB)/s)
170
OS
RHEL
Interconnect
IB EDR
Scheduling Policy (main batch queue)
node-scheduled
Scheduler
LSF
Recommended location for parallel file space
/p/gpfs1
Program
ASC
Class
ATS-2, CORAL-1
Year Commissioned
2018
Compilers

Compilers (CORAL)

Documentation