Job Ephemeral Data Services (JEDS) is a lightweight command-line and Python tool that simplifies the creation of secure, short-lived data service containers for compute jobs on LC clusters. Unlike Persistent Data Services (PDS), JEDS runs only for the duration of a compute resource scheduled job. All data is discarded when the job ends or the service is stopped. JEDS leverages the same LC security integrated and curated container images that PDS uses.
Key Features
- Ephemeral Data: Data is available only for the lifetime of the job; nothing is stored after the job or service ends.
- Supported Services: MariaDB, MongoDB, PostgreSQL, RabbitMQ, Redis.
- Secure by Default: All instances are password protected, use TLS, and comply with LC security policy.
- Flexible Usage: Available as both a CLI tool and a Python library.
- Containerized: Uses Podman to run services in containers.
Getting Started
Prerequisites
- Podman: JEDS uses Podman to manage containers.
If you have never used Podman on LC clusters, run:
enable-podman
Installation
Install the JEDS CLI tool in your Python virtual environment:
python3.11 -m venv venv source venv/bin/activate pip install jeds-cli
Usage
Command Line Interface (CLI)
Start a new ephemeral service instance using the `jeds run` command. For example, to start a RabbitMQ instance:
jeds run rabbitmq
JEDS will print the connection details (host, port, credentials, etc.) after starting the instance.
Supported Services
| Service | Command Example |
|---|---|
| MariaDB | jeds run mariadb |
| MongoDB | jeds run mongodb |
| PostgreSQL | jeds run postgresql |
| RabbitMQ | jeds run rabbitmq |
| Redis | jeds run redis |
Additional Options
JEDS provides additional command-line options to customize the service instance. Refer to the help command for details:
jeds run --help
Python Library
You can also use JEDS as a Python library in your scripts. Example usage:
from jeds.mariadb import MariaDB # create a client to start and stop # a specific mariadb instance mariadb = MariaDB() # start mariadb mariadb.run() # print the connection details print(mariadb.conf) # stop the instance mariadb.stop()
Security
- All JEDS instances are password protected.
- TLS encryption is enabled by default.
- Compliance with LC security policy is enforced.
Service Lifecycle
- Start: Service runs only for the duration of your job.
- Stop: Data is deleted when the job ends or the service is stopped.
Troubleshooting
Common Issues
| Issue | Solution |
|---|---|
| Podman Not Enabled | Run `enable-podman` on LC clusters before using JEDS. |
| Service Not Starting | Check container logs for error messages. |
| Instance Fails Unexpectedly | Check container logs for error messages. |
Debugging with Podman
JEDS uses Podman to run service containers. You can use Podman commands to inspect and debug running or stopped containers:
List JEDS Containers
podman ps -a
This command shows all containers, including those that have exited. Look for containers with names or images related to JEDS.
View Logs for a Container
1. Find the container ID or name from the output of `podman ps -a`.
2. View logs with:
podman logs rabbitmq
This will display the standard output and error logs for the container, which can help diagnose issues.
Example Workflow
podman ps -a podman logs mariadb
Contact & Support
For questions or support, contact lit-help@llnl.gov or lc-hotline@llnl.gov