Why Persistent Data Services (PDS)?🔗

Livermore Computing supports persistent data services such as databases, message broker/queueing systems, and object storage to drive HPC adjacent workflows. In most cases, the significance of offering these services to our users lies in the tooling that these services enable. Various workflow software tools that store metadata on large data sets and the computational steps that generated them will rely on an underlying database and often message broker services. For example, AiiDA requires PostgreSQL and RabbitMQ, whereas Fireworks requires MongoDB.

Livermore Computing’s Workflow Enablement Group (LC-WEG) supports a suite of database services, message brokers, and object storage.

  1. Database services
  2. Message broker & queuing systems
  3. Object Storage

We can help our users configure the workflow tools that depend upon these persistent data services. Examples include AiiDA, Fireworks, and Merlin.

LC WEG deploys and maintains hardware and software infrastructure in support of PDS. Part of this infrastructure includes the containerization of these services, curation of supported containers, integration with LC security best practices and enclaves, and the overall orchestration with OpenShift/Kubernetes. 

How to acquire these resources🔗

To use the database services, message brokers, or object storage mentioned above, resources must first be provisioned.

Please see these docs for more details on how to request or provision these resources.

What to expect🔗

  • Data stored via persistent data services will NOT be backed up by LC (See PDS Deeper Dive for tips on backing up your data).
  • LC can help provision and instantiate the persistent workflow software it offers as well as integrate it with a functional and approved LC security profile. LC does not permanently embed with teams nor administer the applications spun up on the teams' behalf. Applications administration falls to the LC user teams of persistent data services.  
  • PDS will be configured for and be accessible from within a particular network zone. A user running the same workflow tool on both the RZ and CZ, for example, will have separate databases configured for that workflow tool on the RZ and CZ.
  • Regular updates for PDS services will be conducted to meet security compliance requirements. These updates may lead to compatibility issues with workflow tools. User teams will be notified in advance to take necessary administrative actions to accommodate version updates, thereby minimizing security risks. Please note that these updates will be implemented at LC's discretion and may involve specific time constraints for the teams involved before service deactivation. 
  • OAuth token logins will expire 12 hours after being issued, which matches our VPN and kerberos timelines. The OAuth tokens are used to connect to web servers running in the PDS platform.
  • LC offers in-depth assistance with HPC PDS services on a best-effort basis, influenced by factors such as application prevalence among our customers, relevance to diverse teams and projects, availability of in-house expertise, and the various commitments of our subject matter experts (SMEs). Thank you for your understanding as we strive to meet your needs.

For more information on PDS in LC, go to our PDS Deeper Dive page.