LC provides sandboxed versions of the command-line tools for interacting with AI agents. These tools are installed and managed by LC, including default security restrictions that are in line with LLNL policies and best practices. The sandboxing also provides protections for your data to prevent the coding agent from modifying/removing data unexpectedly by restricting the file access permissions available to the coding agent.

LC Currently provides sandboxed versions of codex and claude. The sandboxing is managed by the LC Blackhole tool.

  • Quick Start
  • AI Agent Limitations Imposed by LC's Sandbox
  • Setup a Model Provider
  • Configure the Agent and Sandbox
  • Run the Agent Tool
  • VSCode Extensions

Quick Start

To start using the sandboxed coding agents on LC systems, simply load the codex or claude modules and launch codex or claude, respectively. The wrapper scripts we have in place will guide you through any additional steps you need to get started, including copying a starting configuration file for the coding agent into your home directory.

Examples

module load claude
claude-sandbox

OR

module load codex
codex-sandbox

AI Agent Limitations Imposed by LC's Sandbox

It is highly recommended that LC users deploy AI agents within the protections of the Blackhole sandbox (as described on this page). We expect to require this in the near future. 

File System Access

AI agents are not allowed to operate directly in your home directory, or any other "top-level" user directory (such as /usr/workspace/USERNAME). By default, agents will only be able to access the current working directory and subdirectories.

Network Access

The agent's network access is heavily restricted. Unless approved, it'll only be able to communicate with the LivAI API and Llamame. 

Running Jobs

Currently, sandboxed AI agents may not submit jobs to the cluster's top-level batch scheduler, due to security concerns. Supporting this is on the roadmap.

An agent may be run within a single-node flux allocation, and it can then submit jobs within this allocation.

Setup a Model Provider

The LLM underlying the AI agent can be provided from a number of sources: 

LivAI

Users must request an API key which is used to link their activity to spending and costs associated with running the AI tools. See Getting started with the LivAI API. 

Note that currently on the SCF, there is no LivAI endpoint generally available, but LivIT and AWS are working on it.  For the SCF, LLamaMe is available.  Venado (a LANL Nvidia HPC machine serving OpenAI models) is available if you have an account.

Save your key in a .livai-api-key.txt file in your home directory.

LLamaMe

See LLamaMe Documentation (HPC) for more information on using LLamaMe and getting an API key. The basic steps are:

  1. Visit the LaunchIt website to allocate an LLM service for your use.
    1. CZ: https://launchit.llnl.gov/ or https://lit.llnl.gov
    2. RZ: https://rzlaunchit.llnl.gov/
    3. SCF: https://launchit.llnl.gov/ from a browser on the iSRD network
  2. After signing in, view the catalog and select the "LC LLamaMe API" service. Deploy this to the appropriate workspace (usually zone-username, such as cz-user). Once the service is running, you will be able to get an API key. Full documentation available on the LLamaMe API Service page.
  3. Connect to the deployed LLM via Sandbox. 

Example usage

  1. Store the requested LLamaMe API in a file in your home directory, here called .llamame-api-key.txt 
  2. Export that key as the environment variable $LLAMAME_API_KEY 
  3. Launch your preferred agent tool (such as claude or codex).
module load codex
LLAMAME_API_KEY=$(cat ~/.llamame-api-key.txt) codex-sandbox --config model_provider=llamame-cz --model gemma-4-31b-it

Venado

Configuring for access to Venado-provided models must be made in the Agent-specific configuration files. 

This example works for Codex, and the following should be placed in a .codex/config.toml file:

[model_providers.venado]
name = "Venado GPT" 
base_url = "https://venado.lanl.gov/api/gov-v1" 
env_key = "VENADO_API_KEY"

Configure the Agent and Sandbox

The ability of AI Agents to access files, network, and tools on LC systems is managed by a series of configurations. The default configuration is quite strict and severely limited. Users have some control to relax these limitations. These sandbox (aka blackhole) configurations are specified in yaml files.

  • Project local settings are stored in the working directory in sandbox-config.yaml.  Project-local settings have the highest precedence.
  • User settings can be stored in a user's home directory in .blackhole/config.yaml. These settings take precedence over system settings.

The AI agent tools themselves may require configuration files as well. Please refer to the relevant documentation for each.

Directory Access

AI agents are not allowed to operate directly in your home directory, or any other "top-level" user directory (such as /usr/workspace/USERNAME). Thus, you must launch the tool in a project space. It is recommended that you create a specific directory for an agent to operate in. By default, agents will only be able to access the current working directory and subdirectories.

To allow the agent to access other directories you must specify a configuration file or give the --add flag to the sandbox runtime to specify the additional directory.

A portion of a sandbox-config.yaml configuration file that can be stored in the working directory:

mounts:
  paths:
    - "rw:/usr/workspace/$USER/sandbox/workspace"
    - "ro:/usr/workspace/$USER/sandbox/reference-code:/reference"
    - "try,rw:$HOME/.config"

Configuring mounts.paths 

Path mounts can be expressed in short or long form.

Short form:

[[modifier,]type:]src[:dest]
  • type: Access type is one of ro (read-only), rw (read-write), or tmpfs. Default is ro.
  • modifier: try or del.
  • src: source on the host.
  • dest: destination inside the sandbox. If omitted, dest is set to src.

Short form example:

mounts:
  paths:
    - "/bin"
    - "rw:./:/work"
    - "tmpfs:/run/user"
    - "try:~/.cache"
    - "del:/old/path:/old/path"

Long form example:

mounts:
  paths:
    - source: "$HOME/.codex"
      destination: "$HOME/.codex"
      type: "rw"
      modifier: "try"

Notes:

  • tmpfs mounts use the given path as the destination and do not bind a host source.
  • del removes matching entries from the accumulated mount list.
  • Relative paths are allowed and are resolved before launch.
  • Once policy is locked, later bind mounts must remain under the configured allowed_paths.
  • Environment variables are resolved in this section of the configuration file.

Allowlist rules use root-style matching semantics:

  • /dir allows /dir and everything below it.
  • /dir/* allows entries below /dir, but not /dir itself.
  • Wildcards match one path segment at a time.
  • All symlinks in the path are resolved before matching begins.

Network Access

The agent's network access is heavily restricted. Unless approved, it'll only be able to communicate with the LivAI API and Llamame. If you need to download additional data or push/pull/clone from a git repo, you’ll need to do this yourself before launching the sandbox or from a separate terminal alongside the sandbox.

If you need access to additional network resources:

  1. Email lc-isso@llnl.gov and request an exception for the specific internal domain or IP you need to access from the sandbox
    1. These can be requested on the user or group level

    2. Be sure to include the scope and your justification for access along with the request, for example:

      Please allow 10.0.0.1/32 for my LC group "mygroup" in the RZ because that is a piece of test equipment that I need to query data from for my AI workflow.

  2. Once approved, the configuration changes will be deployed to the clusters the next morning
  3. Update your project configs with the approved network resource.

A portion of a  sandbox-config.yaml configuration file that can be stored in the working directory:

network:
  allow_ips:
    - "your-approved-subdomain.llnl.gov"
    # IP addresses also work

MPI Access

The sandbox can run MPI applications on a single node. This is done by running the harness with the --mpi option, which launches a flux instance in the background (when necessary). 

Once the tool is running on an allocated compute node, you may need to load the flux_wrappers module in order to access slurm commands. You should do this before running the sandboxed agent:

$ salloc -N 1 -q pdebug -t 01:00:00 --exclusive
# preload the flux slurm wrappers
$ module load flux_wrappers
$ /collab/usr/global/tools/sandbox/codex-sandbox --mpi

NOTE Some projects might be set up to use an absolute path for srun   (e.g., /usr/bin/srun). If so, you might need to override the path. For example, in CMake you can set -DMPI_EXECUTABLE=$(which srun) once the flux wrappers are loaded

Run the Agent Tool

LC currently supports two AI Agent harnesses:

  • claude (developed by Anthropic)
  • codex (developed by OpenAI)

Users can access these tools via LC modules

module load claude
claude-sandbox

OR

module load codex
codex-sandbox

It is recommended that you allocate a compute node when utilizing an AI agent. This prevents the login nodes from becoming overburdened by many users running resource-intensive AI agent workloads. To request an interactive allocation 

  • On a slurm system: salloc -N 1 -p pdebug -t 01:00:00 --exclusive
  • On a flux system: flux alloc -N 1 -q pdebug -t 1h --exclusive

These commands put your shell inside the allocation on the compute node. 

VSCode Extensions

If you want to use the Codex or Claude Code extensions with VS Code over remote SSH to an LC machine, you'll need to configure them to use our sandbox wrappers.

1. Open the VS Code JSON settings editor. Press Command-Shift-P or Ctrl-Shift-P and enter the following command into the command bar:

@command:workbench.action.openSettingsJson

2. Add the appropriate config parameters to the top-level object in your settings JSON:

"claudeCode.claudeProcessWrapper": "/collab/usr/global/tools/sandbox/claude-vscode",
"claudeCode.disableLoginPrompt": true,
"chatgpt.cliExecutable": "/collab/usr/global/tools/sandbox/codex-vscode"

Codex & VSCode Caveats

Codex doesn't launch its VSCode extension in your project's directory, so you need to set up a sandbox configuration file before launching the extension so the agent can read your project's files. The sandbox launcher expects the Codex VSCode sandbox config to be located at ~/.codex/vscode-config.yaml. This file should minimally contain the paths you'll want to use with VSCode.

The Codex agent also won't parse any project-level sandbox configfiles you have at launch. You can add these as includes in your VSCode sandbox config file, but be mindful that best practice is to grant only the minimum permissions needed for the agent.

Contacts

Any questions or feedback can be directed to:

  • LC Hotline – (925)-422-4531 – lc-hotline@llnl.gov. 
  • Blackhole developer mailing list - lc-sandbox@llnl.gov
  • WSC Coding Agents channel on Teams
  • (Open side) LC Public channel on the ASC-Trilab Mattermost (or Blackhole Sandbox channel on LC internal Mattermost)

For any issues with LivAI or the API services (API errors, etc.), please reach out directly to LivAI through ServiceNow. Unfortunately, LC has very limited ability to assist with these issues.