Parallel File Systems

This page contains information on both the Lustre and GPFS file systems. Most LC machines use Lustre, an open source parallel file system. However, the Coral machines came with GPFS instead of Lustre, so now we have both file systems in house. GPFS is IBM's parallel file system (currently known as Spectrum Scale).


Quotas will be in place to monitor and regulate both the total amount of data and the total number of files used by a particular user. The implementation will include both hard quotas and soft quotas. A hard quota will prevent new file writes from succeeding until the user has freed up space below the quota threshold. The soft quota is a lower threshold, set at 90% of the hard threshold and is equal to the user's quota. When the soft quota is hit, it starts a grace period for the user to clear space or files before having all writes blocked. During this grace period, I/O performance will likely be throttled. Once that probationary period is exceeded and the user has not yet reduced usage below their quota, the soft quota becomes a hard quota and new writes will fail.

There are 3 levels of quotas for users, as shown in Table 1 below. Movement of a user from Tier 1 to Tier 2 may be requested from and implemented by the LC Hotline and are permanent. For those needing additional capacity or number of files, they can provide justication for a custom amount to their account coordinator who can then implement the change for up to six months.

The quota limits are displayed in the following table:

LC Quota Tiers

Quota Tier Capacity (TB) # Files Grace Period (days)
Soft Hard Soft Hard
1 20 22 1M 1.1M 10
2 75 82 15M 16.7M 10
3 Quotas set per amount justified 10

GPFS File System

In all cases, the GPFS filesystems exist only on their individual clusters.

Zone Machine Mount Capacity (PiB)


Ray /p/gscratchr 1.3
Lassen /p/gpfs1 24
RZ RZManta /p/gscratchrzm 0.43
RZAnsel /p/gpfs1 1.5
SCF Shark /p/gscratchs 0.43
Sierra /p/gpfs1 140


Lustre File System 

NOTE Large systems such as Quartz, Jade, RZTopaz, and Zin have significantly more bandwidth to Lustre.

CZ File System Mount Capacity (PiB)
lustre1 /p/lustre1 (/p/czlustre1 on rzslic) 15
lustre2 /p/lustre2 (/p/czlustre2 on rzslic) 15
lustre3 /p/lustre3 (only on oslic, rzslic, surface, pascal) 4.9
RZ File System
Capacity (PiB)
lustre1 /p/lustre1 7.5
SCF File System Mount Capacity (PiB)
lustre1 /p/lustre1 15
lustre2 /p/lustre2 15
SNSI File System Mount Capacity (PiB)
lustre1 /p/lustre1 2.5