Parallel File Systems

This page contains information on both the Lustre and GPFS file systems. Most LC machines use Lustre, an open source parallel file system. However, the Coral machines came with GPFS instead of Lustre, so now we have both file systems in house. GPFS is IBM's parallel file system (currently known as Spectrum Scale).
 

GPFS File System

In all cases, the GPFS filesystems exist only on their individual clusters.

ZoneMachineMountCapacity (PiB)

CZ

Ray/p/gscratchr1.3
Lassen/p/gpfs124
RZRZManta/p/gscratchrzm0.43
RZAnsel/p/gpfs11.5
SCFShark/p/gscratchs0.43
Sierra/p/gpfs1140

 

Lustre File System 

NOTE Large systems such as Quartz, Jade, RZTopaz, and Zin have significantly more bandwidth to Lustre.

CZ File SystemMountCapacity (PiB)
lustre1/p/lustre1 (/p/czlustre1 on rzslic)15
lustre2/p/lustre2 (/p/czlustre2 on rzslic)15
lustre3/p/lustre3 (only on oslic, rzslic, surface, pascal)4.9
RZ File System
Mount
Capacity (PiB)
lustre1/p/lustre17.5
SCF File SystemMountCapacity (PiB)
lustre1/p/lustre115
lustre2/p/lustre215
SNSI File SystemMountCapacity (PiB)
lustre1/p/lustre12.5

Quotas

Quotas will be in place to monitor and regulate both the total amount of data and the total number of files used by a particular user. The implementation will include both hard quotas and soft quotas. A hard quota will prevent new file writes from succeeding until the user has freed up space below the quota threshold. The soft quota is a lower threshold, set at 90% of the hard threshold and is equal to the user's quota. When the soft quota is hit, it starts a grace period for the user to clear space or files before having all writes blocked. During this grace period, I/O performance will likely be throttled. Once that probationary period is exceeded and the user has not yet reduced usage below their quota, the soft quota becomes a hard quota and new writes will fail.

There are 3 levels of quotas for users, as shown below. Movement of a user from Tier 1 to Tier 2 may be requested from and implemented by the LC Hotline and are permanent. For those needing additional capacity or number of files, they can provide justification for a custom amount to their account coordinator who can then implement the change for up to six months.

Quotas for File Systems and Storage

LC Storage TypeTier 1Tier 2Tier 3

Snap-
shots?

Back up to tape?
Parallel File Systems

   /p/lustre* (Lustre)

   /p/gpfs*
(GPFS - pending deployment)

   /p/vast*
(Vast - pending deployment)
20TB/1M75TB/10M Contact LC Hotline to request increaseContact LC Hotline to initiate conversation with Livermore Computing and programmatic stakeholdersNoNo
NAS/NFS Project

  /usr/workspace,
  /usr/WS*

  /usr/gapps

  /usr/gdata

 


2TB/10M


10GB


10GB
Contact LC Hotline to initiate conversation with Livermore Computing and programmatic stakeholders


Yes


Yes


Yes



No


Yes


Yes
Home Directory

  /g/g*


24GB


Yes


Yes
Archive (HPSS)

  /users/u*


300TB


No


No

these quotas are per directory, not per user as in all other cases