LC is providing object storage compatible with Amazon Simple Storage Service (S3)
LC is providing an object-based storage system backed by NetApp's Storage Grid Product. To learn more about object based storage please visit: Wikipedia
Purpose of these docs
Please note this isn't a full tutorial! We want to show you some basic commands to get you started. Please refer to the s3api documentation for more info on the commands you can use with the AWS CLI.
Basic Setup🔗
After allocating S3 object storage via LaunchIT, you will need to configure a profile using the AWS command line interface (CLI).
In LC version 2 of the AWS CLI installed at /usr/global/awscli/aws. You can either run the aws command with the full path or add this directory to your $PATH variable, for example:
export PATH=$PATH:/usr/global/awscli
You can use the following syntax to use the AWS CLI to begin creating a profile called lc-objstore:
aws configure --profile lc-objstore
After running this command, you'll be prompted to provide the access-key and secret-key provided to you with your object storage account. Below is an example of the prompts you'll see. The values my-access-key and my-secret-key should be replaced with your information, and you can simply hit Enter to pass the prompts for "Default region name" and "Default output format":
AWS Access Key ID [None]: my-access-key AWS Secret Access Key [None]: my-secret-key Default region name [None]: [enter] Default output format [None]: [enter]
After completing the above, update your profile to use a locally customized certificate bundle by running
aws configure --profile lc-objstore set ca_bundle /etc/pki/tls/cert.pem
Note that you should replace lc-objstore with the name of the profile you've created!
Below are examples both at the shell and using the boto3 python library.
S3 at the shell🔗
The S3 provisioned for you should come with an existing bucket. In the sections below, you'll see how to list existing buckets, modify buckets (uploading and deleting objects from buckets), query buckets (listing and downloading objects from a bucket), and create new buckets.
List a bucket🔗
aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov list-buckets
Note: replace lc-objstore with the name of the profile you've created!
After this, you'll see output like
{ "Owner": { "DisplayName": "<username>", "ID": "<Account ID>" }, "Buckets": [ { "CreationDate": "2022-01-22T00:29:03.070Z", "Name": "cz-my-bucket" } ] }
Note the name of your bucket from the output you see. Here the bucket name is cz-my-bucket.
Upload an object🔗
In the following command, you upload an object to be named my-hosts (from the --key flag) to an existing bucket cz-my-bucket under the profile lc-objstore. The content of my-hosts is /etc/hosts specified by the --body flag and info on the object is specified by --metadata:
aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov put-object --key my-hosts --body /etc/hosts --metadata '{"hosts":"local-cluster"}' --bucket cz-my-bucket
Note: if you try this, replace flag values with your info. For example, replace lc-objstore with the name of the profile you've created and replace cz-my-bucket with the name of your bucket!
This returns something like
'{"hosts":"local-cluster"}' { "ETag": "\"8b14f67903beb8db9d332a552aa60c3f\"" }
List objects in a bucket🔗
aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov list-objects --bucket cz-my-bucket
Note: replace lc-objstore and cz-my-bucket with your information!
Output here will look like
{ "Contents": [ { "LastModified": "2022-01-22T00:34:15.065Z", "ETag": "\"8b14f67903beb8db9d332a552aa60c3f\"", "StorageClass": "STANDARD", "Key": "my-hosts", "Owner": { "DisplayName": "<username>", "ID": "<Account ID>" }, "Size": 33340 } ] }
Download an object🔗
You can download an object via
aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov get-object --key my-hosts downloaded-file --bucket cz-my-bucket
Note: replace lc-objstore and cz-my-bucket with your information!
which returns a message like
{ "AcceptRanges": "bytes", "ContentType": "binary/octet-stream", "LastModified": "Sat, 22 Jan 2022 00:34:15 GMT", "ContentLength": 33340, "ETag": "\"8b14f67903beb8db9d332a552aa60c3f\"", "Metadata": { "hosts": "local-cluster" } }
To check the downloaded file,
$ tail -1 downloaded-file 172.19.3.171 oslic21-san0 oslic21.llnl.gov oslic21
Delete an object🔗
aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov delete-object --key my-hosts --bucket cz-my-bucket
Note: replace lc-objstore and cz-my-bucket with your information!
Create a bucket🔗
To create additional buckets,
aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov create-bucket --bucket cz-my-bucket
Note: replace lc-objstore and cz-my-bucket with your information!
After running this, you should see something like
{ "Location": "/cz-my-bucket" }
printed to stdout.
Note that if you choose a bucket name that has already been used by another user, including cz-my-bucket as above, you'll see
An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
So, you'll have to get more creative!
S3 with python boto3🔗
https://boto3.amazonaws.com/v1/documentation/api/latest/index.html
Create a virtual environment and install boto3🔗
python3 -m venv boto-venv source boto-venv/bin/activate pip install -U pip pip install boto3
Start python, import boto3 and set up the connection details
$ python >>> import boto3 >>> import boto3.session >>> session = boto3.session.Session(profile_name='lc-objstore') >>> endpoint = 'https://osgha.llnl.gov' >>> s3 = session.resource(service_name='s3', endpoint_url=endpoint) >>> client = s3.meta.client
Remember to replace lc-objstore with your profile name!
List buckets
>>> for bucket in s3.buckets.all(): ... print(bucket.name) ... mybucket
Here mybucket is the name of the bucket returned.
List files in a bucket
>>> for o in s3.Bucket('mybucket').objects.all(): ... print("Key: " + o.key) ... print("Size: " + str(o.size)) ... ... Key: my-hosts Size: 40246
Remember to replace mybucket with the name of your bucket!