Skip to content

Persistent Data Services (PDS) at Livermore Computing

Why Persistent Data Services (PDS)?

Livermore Computing has recently begun supporting persistent data services such as databases, message broker/queueing systems, and object storage. In most cases, the significance of offering these services to our users lies in the tooling that these services enable. Various workflow software tools that store metadata on large data sets and the computational steps that generated them will rely on an underlying database and often message broker services. For example, AiiDA requires PostgreSQL and RabbitMQ, whereas Fireworks requires MongoDB.

Where are we (headed)?

Livermore Computing’s Workflow Enablement Group (LC-WEG) supports a suite of DB services, message brokers, and object storage.

  1. Database services
  2. Message broker & queuing systems
  3. Object Storage

We also help our early users to configure the workflow tools that depend upon these PDS. So far we have users using AiiDA, Fireworks, and Merlin.

LC WEG deploys and maintains hardware and other infrastructure in support of PDS, including Clowder, a cluster on the CZ.

How to get help

Please reach out to the Livermore Computing Hotline if you would like to request PDS for your application (using the template below) or if you are experiencing problems with any of our PDS.

You can email lc-hotline@llnl.gov or call (925) 422-4531.

PDS request:

I would like to use <AiiDA>, a <workflow tool>, in support of my work in <PLS>. My programmatic supervisor is <Joe Smith - smith0>. 

<AiiDA> relies on the persistent data services (PDS) <PostgreSQL> and <RabbitMQ>. Could you please set me up on these PDS?

Zone: <CZ>     
LC Username: <janeh>
Estimate of data scale: <Tens of GB>

Thank you!
<Jane>

What to expect

  • Data stored via persistent data services will NOT be automatically backed up. (See PDS Deeper Dive for help on backing up your data.)
  • PDS will be configured for and be accessible from within a particular network zone. A user running the same workflow tool on both the RZ and CZ, for example, will have separate databases configured for that workflow tool on the RZ and CZ.
  • Regular updates will occur for PDS services, as part of security compliance requirements. This may create compatibility issues with workflow tools.
  • OAuth token logins will expire 12 hours after being issued, which matches our VPN and kerberos timelines. The OAuth tokens are used to connect to web servers running in the PDS platform.

For more information on PDS in LC, go to our PDS Deeper Dive page.