Persistent Data Services (PDS) at Livermore Computing¶
Why Persistent Data Services (PDS)?¶
Livermore Computing has recently begun supporting persistent data services such as databases, message broker/queueing systems, and object storage. In most cases, the significance of offering these services to our users lies in the tooling that these services enable. Various workflow software tools that store metadata on large data sets and the computational steps that generated them will rely on an underlying database and often message broker services. For example, AiiDA requires PostgreSQL and RabbitMQ, whereas Fireworks requires MongoDB.
Where are we (headed)?¶
Livermore Computing’s Workflow Enablement Group (LC-WEG) supports a suite of DB services, message brokers, and object storage.
- Database services
- Message broker & queuing systems
- Object Storage
- Storage Grid (Uses S3 Buckets)
LC WEG deploys and maintains hardware and other infrastructure in support of PDS, including Clowder, a cluster on the CZ.
How to acquire these resources¶
To use any of the database services, message brokers, or object storage mentioned above, resources must be provisioned for you.
Please see these docs for more details on how to request or provision these resources.
What to expect¶
- Data stored via persistent data services will NOT be automatically backed up. (See PDS Deeper Dive for help on backing up your data.)
- PDS will be configured for and be accessible from within a particular network zone. A user running the same workflow tool on both the RZ and CZ, for example, will have separate databases configured for that workflow tool on the RZ and CZ.
- Regular updates will occur for PDS services, as part of security compliance requirements. This may create compatibility issues with workflow tools.
- OAuth token logins will expire 12 hours after being issued, which matches our VPN and kerberos timelines. The OAuth tokens are used to connect to web servers running in the PDS platform.
For more information on PDS in LC, go to our PDS Deeper Dive page.