The availability, accessibility, and reliability of scientific data are critical to the success of the missions that rely on our world-class computers. In addition to burst buffers and tape archives, parallel file systems are a key part of the storage triad that keeps our data safe and accessible.
Parallel File Systems
There are two parallel file systems currently deployed in LC: Lustre and GPFS. While Lustre systems support many of the compute platforms, GPFS supports the CORAL-1 machines such as Lassen and Sierra.
LC’s Lustre Development Team develops and supports the primarily Lustre-based high-performance parallel file systems used for LC’s commodity technology systems (CTS). Lustre is a parallel distributed file system that works particularly well with large-scale cluster computing, the workhorses of parallel computing at the Lab. Lustre is also open source, enabling LC developers to tailor it to specific systems and the needs of those systems’ users. The team’s work helps ensure efficient, secure, and reliable computation and data storage on an array of platforms. It is also influencing industry standards in file systems.
LC’s CORAL machines (Sierra, Lassen, Shark, RZManta, and Ray) rely on IBM’s General Parallel File System (GPFS) (also known as Spectrum Scale) for high-performance storage during the processing of large scale simulations. GPFS is a high performance shared-disk file management solution that provides fast, reliable access to data from multiple servers. Applications can readily access files using standard file system interfaces, and the same file can be accessed concurrently from multiple servers and protocols.
For more on computer storage and file systems:
• User Portal File Systems Page