Skip to content

Object Storage

LC is providing early access to object storage compatible with Amazon Simple Storage Service (S3)

LC is providing an object-based storage system backed by NetApp's Storage Grid Product. To learn more about object based storage please visit: Wikipedia

Purpose of these docs

Please note this isn't a full tutorial! We want to show you some basic commands to get you started. Please refer to the s3api documentation for more info on the commands you can use with the AWS CLI.

Basic Setup

After having S3 object storage created and set up for you, you will need to configure a profile using the AWS command line interface (CLI).

TOSS 3 has version 1 of the AWS CLI installed. You can use the following syntax to use the AWS CLI to begin creating a profile called lc-objstore:

aws configure --profile lc-objstore

After running this command, you'll be prompted to provide the access-key and secret-key provided to you with your object storage account. Below is an example of the prompts you'll see. The values my-access-key and my-secret-key should be replaced with your information, and you can simply hit Enter to pass the prompts for "Default region name" and "Default output format":

``` AWS Access Key ID [None]: my-access-key

AWS Secret Access Key [None]: my-secret-key

Default region name [None]: [enter]

Default output format [None]: [enter] ```

After completing the above, apdate your profile to use a locally customized certificate bundle by running

aws configure --profile lc-objstore set ca_bundle /etc/pki/tls/cert.pem

Note that you should replace lc-objstore with the name of the profile you've created!

Below are examples both at the shell and using the boto3 python library.

S3 at the shell

The S3 provisioned for you should come with an existing bucket. In the sections below, you'll see how to list existing buckets, modify buckets (uploading and deleting objects from buckets), query buckets (listing and downloading objects from a bucket), and create new buckets.

List a bucket

aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov list-buckets

Note: replace lc-objstore with the name of the profile you've created!

After this, you'll see output like

{ "Owner": { "DisplayName": "<username>", "ID": "<Account ID>" }, "Buckets": [ { "CreationDate": "2022-01-22T00:29:03.070Z", "Name": "cz-my-bucket" } ] }

Note the name of your bucket from the output you see. Here the bucket name is cz-my-bucket.

Upload an object

In the following command, you upload an object to be named my-hosts (from the --key flag) to an existing bucket cz-my-bucket under the profile lc-objstore. The content of my-hosts is /etc/hosts specified by the --body flag and info on the object is specified by --metadata:

aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov put-object --key my-hosts --body /etc/hosts --metadata '{"hosts":"local-cluster"}' --bucket cz-my-bucket

Note: if you try this, replace flag values with your info. For example, replace lc-objstore with the name of the profile you've created and replace cz-my-bucket with the name of your bucket!

This returns something like

'{"hosts":"local-cluster"}' { "ETag": "\"8b14f67903beb8db9d332a552aa60c3f\"" }

List objects in a bucket

aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov list-objects --bucket cz-my-bucket

Note: replace lc-objstore and cz-my-bucket with your information!

Output here will look like

{ "Contents": [ { "LastModified": "2022-01-22T00:34:15.065Z", "ETag": "\"8b14f67903beb8db9d332a552aa60c3f\"", "StorageClass": "STANDARD", "Key": "my-hosts", "Owner": { "DisplayName": "<username>", "ID": "<Account ID>" }, "Size": 33340 } ] }

Download an object

You can download an object via

aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov get-object --key my-hosts downloaded-file --bucket cz-my-bucket

Note: replace lc-objstore and cz-my-bucket with your information!

which returns a message like

{ "AcceptRanges": "bytes", "ContentType": "binary/octet-stream", "LastModified": "Sat, 22 Jan 2022 00:34:15 GMT", "ContentLength": 33340, "ETag": "\"8b14f67903beb8db9d332a552aa60c3f\"", "Metadata": { "hosts": "local-cluster" } }

To check the downloaded file,

$ tail -1 downloaded-file 172.19.3.171 oslic21-san0 oslic21.llnl.gov oslic21

Delete an object

aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov delete-object --key my-hosts --bucket cz-my-bucket

Note: replace lc-objstore and cz-my-bucket with your information!

Create a bucket

To create additional buckets,

aws s3api --profile lc-objstore --endpoint-url https://osgha.llnl.gov create-bucket --bucket cz-my-bucket

Note: replace lc-objstore and cz-my-bucket with your information!

After running this, you should see something like

{ "Location": "/cz-my-bucket" }

printed to stdout.

Note that if you choose a bucket name that has already been used by another user, including cz-my-bucket as above, you'll see

An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

So, you'll have to get more creative!

S3 with python boto3

https://boto3.amazonaws.com/v1/documentation/api/latest/index.html

Create a virtual environment and install boto3

python3 -m venv boto-venv source boto-venv/bin/activate pip install -U pip pip install boto3

Start python, import boto3 and set up the connection details

$ python
>>> import boto3
>>> import boto3.session
>>> session = boto3.session.Session(profile_name='lc-objstore')
>>> endpoint = 'https://osgha.llnl.gov'
>>> s3 = session.resource(service_name='s3', endpoint_url=endpoint)
>>> client = s3.meta.client

Remember to replace lc-objstore with your profile name!

List buckets

>>> for bucket in s3.buckets.all():
...   print(bucket.name)
...
mybucket

Here mybucket is the name of the bucket returned.

List files in a bucket

>>> for o in s3.Bucket('mybucket').objects.all():
...   print("Key: " + o.key)
...   print("Size: " + str(o.size))
...
...
Key: my-hosts
Size: 40246

Remember to replace mybucket with the name of your bucket!