What is AiiDA?🔗
AiiDA is an open-source workflow manager geared towards automating complex calculations. At it's core it is a generic framework which handles the submission of calculations, manages the data storage of calculation results, and enables complex error handling to name a few features.
AiiDA requires several other services to run such as a Postrgres database and a RabbitMQ service. Both of these can be found at LaunchIT. Before starting your AiiDA installation process you should initiate both of these services and keep the information readily available.
Getting started with AiiDA🔗
The original development of AiiDA developed around the idea of allowing a personal laptop act as the central repository. It would then be able to log into other computers/servers to submit jobs and retrieve after they had finished. With the advancedment of firewalls and security, especially at the lab, it is not as feasible to configure AiiDA with this type of installation. The following instructions will outline the total installation process on LC machines at the lab.
Cloning and basic AiiDA Setup🔗
There are two ways that you can install the core AiiDA package into your system. For either case, it is highly recommended to create a python or conda environment to keep your AiiDA installation separate to prevent breaking possible dependencies. This can be done with
python -m venv /path/to/environment/aiida source /path/to/environment/aiida/bin/activate
or
conda create -n aiida conda activate aiida
Once you have your environment initialized, you can install AiiDA through pip
pip install aiida
or download and and install from the AiiDA git repository.
git clone https://github.com/aiidateam/aiida-core.git cd aiida-core git checkout v2.6.2 # Decide if there is a more recent version you will use. pip install -e .
Anytime that you will be working with AiiDA you will need to activate this environment to gain access to the AiiDA commands. Activating your environment adds the name of your environment before the command prompt. It also changes the default binaries for python and pip to be those found in your environment.
Creating a profile🔗
The first thing you'll need to do is create a profile that will be associated with all the simulations launched through AiiDA. The profile will control which Postgres database and RabbitMQ service you will be connecting to. This can be used to separate projects if needed.
The main command to interact with AiiDA is `verdi`. This command should be available to you after installing the AiiDA package. You can list all possible commands with `verdi --help`.
There are several ways to setup a profile. The first is typing
verdi profile setup core.psql_dos
This will prompt you for relative information to set up your profile. A somewhat easier method would be to simply pass a yaml file that the command will read to create your profile.
verdi profile setup core.psql_dos --config profile.yaml
This file should look something like this:
non_interactive: y profile_name: <profile_name> email: <email> first_name: <First> last_name: <Last> institution: LLNL set_default: true database_engine: core.psql_dos database_hostname: <hostname> # Postgres LaunchIT database_port: <port> # Postgres LaunchIT database_name: <name> # Postgres LaunchIT database_username: <username> # Postgres LaunchIT database_password: <password> # Postgres LaunchIT broker_protocol: amqps broker_username: <username> # RabbitMQ LaunchIT broker_password: <password> # RabbitMQ LaunchIT broker_host: <hostname> # RabbitMQ LaunchIT broker_port: <port> #RabbitMQ LaunchIT broker_virtual_host: aiida repository: /path/to/where/you/want/the/repository
The last line with `repository` should be set to space with a large amount of storage. Your /usr/workspace can work for this. At this point you should be able to see that you have a profile created with `verdi profile list` which should return your chosen username.
For AiiDA to run on LC machines, you will need to edit one more file. There will be a hidden folder named `.aiida` either where you specified your repository or in your home folder. Inside of the folder you will need to edit a file named `config.json` and add broker_parameters after the broker_virtual_host field.
"broker_port": 32200,
"broker_virtual_host": "aiida",
"broker_parameters": {
"no_verify_ssl": "1",
"cafile": "/etc/pki/tls/cert.pem"
},
At this point you can test the status of the AiiDA installation by typing `verdi status`. The only thing that should not be initiated at this point is the daemon which interacts with the RabbitMQ server. This can be started with
verdi daemon start
By default it starts with a single daemon. You can increase the number by setting a number at the end of the command or by increasing or decreasing the current number with
verdi daemon incr 1 verdi daemon decr 1
Additionally, a nice with to interact with the python api is to use
verdi shell
which essentially initiates an ipython instance with many of the AiiDA modules pre-loaded.
Set up and configure a computer🔗
Set up computer🔗
A computer within the AiiDA context is where calculations will be ran. This can be any machine that you can ssh into and submit jobs. To give an example, we will create a computer for the Ruby server here at the lab. When setting up a computer. You will want to create a directory where all the simulations will be ran. Ideally, this would be some kind of scratch file system that has plenty of space for your simulations. For Ruby we have the Lustre file system which can be found at "/p/lustre1/<username>". I would recommend creating a folder in that space named AiiDA or something that will prompt you not to delete it in the future.
Similarly to the profile setup process, we can provide a yaml file to simplify the installation process. This can be done with
verdi computer setup --config ruby.yaml
where ruby.yaml is the file that will contain the following settings
label: ruby
hostname: "ruby.llnl.gov" # Make sure you can ssh to this before installation by typing `ssh ruby.lln.gov`
description: "Ruby server at LLNL"
transport: core.ssh
scheduler: "core.slurm"
work_dir: "/p/lustre1/<username>/AiiDA" # You will need to confirm which lustre folder you are a part of.
mpirun_command: "srun -n {tot_num_mpiprocs}"
mpiprocs_per_machine: "56"
prepend_text: ""
append_text: " "
shebang: "#!/bin/bash"
Once the initial step has been taken to setup the computer you will need to finish configuring the computer with
verdi computer configure core.ssh configure_ruby.yml
where configure_ruby.yaml contains
username: "<lc_username>" port: 22 look_for_keys: true key_filename: "/g/g1/<username>/.ssh/id_ecdsa" # Path to where your SSH key is at timeout: 60 allow_agent: true proxy_command: "" compress: true gss_auth: false gss_kex: false gss_deleg_creds: false gss_host: "ruby" load_system_host_keys: true key_policy: "RejectPolicy" use_login_shell: true safe_interval: 10.0
Test computer🔗
To make sure this computer is configured correctly, execute and make sure all tests pass:
verdi computer test ruby
Make sure that all tests pass at this point. Some common issues include needing to modify your .bashrc file to allow for non-interactive sessions. One way to do this is to prepend the following text to your .bashrc file.
if [[ $- != *i* ]] ; then
return
fi
Configure code🔗
At this point AiiDA can connect to your database/RabbitMQ service and connect to the different servers to which you are wanting to submit calculations. The next step in the process is to configure a code executable so that AiiDA knows where it is located. For an example, we will show how INQ can be configured to run within AiiDA.
Install plug in for INQ🔗
AiiDA works by relying on plugins to specify how a code will interact with the AiiDA framework. Many of the popular code packages already have a community that supports a plug-in and can found at the AiiDA plug-in registry.
Before installing aiida-inq, running `verdi plugin list aiida.calculations` will show something like:
verdi plugin list aiida.calculations Registered entry points for aiida.calculations: * arithmetic.add * core.transfer * templatereplacer Report: Pass the entry point as an argument to display detailed information
To install the plug-in for INQ, execute the following commands:
git clone https://github.com/LLNL/aiida-inq cd aiida-inq pip install -e .
After installing aiida-inq, the plugin list should be updated to show the new calculations that can be performed.
Set up INQ code🔗
To set up INQ with a script, create file code.yml with the following content
label: inq description: "INQ code from LLNL" default_calc_job_plugin: "inq.inq" filepath_executable: "/bin/bash" computer: ruby use_double_quotes: false with_mpi: true prepend_text: "" append_text: ""
You might notice that the path to the executable has been set to "/bin/bash". The AiiDA implementation for INQ works by creating a bash script. Inside of that script it will make reference to the INQ binary. With that being the case, make sure that the INQ binary can be found in your $PATH. This can be done by appending the following line to your .bashrc file
export PATH="/path/to/inq/binary:$PATH"
Run an INQ calculation🔗
Running a job from a script🔗
One of the easiest ways to launch a job within AiiDA is through simple python scripts. An example script, launch.py, is shown below
from aiida import load_profile
from aiida.orm import load_code, Dict
from aiida.plugins import CalculationFactory, DataFactory
from aiida.engine import run,submit
from ase.build import bulk
# Initiate the default profile
load_profile()
# Get the calculator from AiiDA
InqCalculation = CalculationFactory('inq.inq')
# Find the code you will use for the calculation
code = load_code('inq@ruby')
# Create a structure
StructureData = DataFactory('core.structure')
atoms = bulk('Si', crystalstructure='diamond', a=5.43)
atoms.positions
structure = StructureData(ase=atoms)
inputs = {
'code': code,
'structure': structure,
'parameters' : Dict(dict={
'electrons': {
'cutoff': '35.0 Ha',
'extra-states': 3
},
'kpoints': {
'gamma': '',
'insert': '-0.5 -0.5 -0.5 0.0'
},
'ground-state': {
'tolerance': 1e-8
},
'run': {
'ground-state': ''
}
}),
'metadata': {
#'dry_run': True, # If uncommented, will only create the files.
#'store_provenance': False, # Will not store any of the provenance in the database.
'options': {
'resources': {
'tot_num_mpiprocs': 4
}
}
}
}
# Will show detailed results
run(InqCalculation, **inputs)
# Comment the previous line and use the following if you want to get the pk value to follow along with.
#calc = submit(InqCalculation, **inputs)
#print(f'Created calculation with PK={calc.pk}')
which can then be run via
verdi run launch.py
You can again learn the pk of this number by running `verdi process list` or `verdi process list -a` to show all calculations. Check the help documentation, `verdi process list --help`, to see all the possible options.
Checking job status in AiiDA🔗
If you run verdi process list, current calculation will show up at the bottom of the list, with its identifying “pk” number in the leftmost column:
verdi process list PK Created Process label Process State Process status ---- --------- --------------- --------------- ---------------- 3339 13m ago InqCalculation ⏹ Finished [0]
You can then take that number and run verdi process show <pk> to learn more about the process.
Property Value
----------- -------------------------------------------------------------
type InqTDDFTWorkChain
state Finished [11] The process did not register a required output.
pk 3339
uuid c9d7fde2-12bf-4901-9893-3954695ff78c
label
description
ctime 2024-10-01 17:58:04.092209+00:00
mtime 2024-10-01 17:58:18.093927+00:00
Inputs PK Type
------------------ ---- -------------
gs
inq
code 1 InstalledCode
structure 3331 StructureData
parameters 3332 Dict
clean_workdir 3333 Bool
max_iterations 3334 Int
tddft
inq
code 1 InstalledCode
structure 3331 StructureData
parameters 3335 Dict
clean_workdir 3336 Bool
max_iterations 3337 Int
clean_workdir 3338 Bool
structure 3331 StructureData
Outputs PK Type
----------------- ---- -------------
output_parameters 3354 Dict
output_structure 3353 StructureData
Called PK Type
------------ ---- ----------------
Ground_State 3341 InqBaseWorkChain
TDDFT 3348 InqBaseWorkChain
Log messages
---------------------------------------------
There are 3 log messages for this calculation
Run 'verdi process report 3339' to see them
Troubleshooting a job🔗
Once you have the pk number of a job, you can learn more about the calculation and its outputs via
verdi process show <pk> verdi process report <pk> verdi calcjob gotocomputer <pk>
The last option takes you to a directory where you’ll see the outputs from the job and the script actually submitted to slurm to run the desired calculations.
If necessary, you can kill a process with verdi process kill <pk>.
Getting results🔗
To look at the results of the calculation you can either write a script or start a verdi shell instance. The following code will get the results of the previous calculation.
verdi shell node = load_node(3339) outputs = node.outputs.output_parameters.get_dict() print(outputs)
Conclusion🔗
Hopefully this brief tutorial is enough to get you started running calculations using AiiDA on the LC servers at LLNL. For further documentation we highly suggest going over the documentation and tutorials that AiiDA has on their website.