Skip to content

Object Storage

LC is providing early access to an Amazon Simple Storage Service (S3)

LC is providing compatible object based storage system backed by NetApp's Storage Grid Product. To learn more about object based storage please visit: Wikipedia

Basic Setup

Below are examples both at the shell and using the boto3 python library.

Both need an AWS config file setup.

TOSS 3 has version 1 of the AWS CLI installed, it is configured like below, which creates a profile call lc-objstore, using the access-key and secret-key provided to you with your object storage account.

aws configure --profile lc-objstore

AWS Access Key ID [None]: my-access-key

AWS Secret Access Key [None]: my-secret-key

Default region name [None]: [enter]

Default output format [None]: [enter]

Update your profile to use locally customized certificate bundle

aws configure --profile lc-objstore set ca_bundle /etc/pki/tls/cert.pem

S3 at the shell

List a bucket

$ aws s3api --profile lc-objstore --endpoint-url https://sgusrgw1.llnl.gov:8082 list-buckets

Upload an object

$ aws s3api --profile lc-objstore --endpoint-url https://sgusrgw1.llnl.gov:8082 put-object --bucket mybucket --key my-hosts --body /etc/hosts --metadata '{"hosts":"local-cluster"}'

List objects in a bucket

$ aws s3api --profile lc-objstore --endpoint-url https://sgusrgw1.llnl.gov:8082 list-objects --bucket mybucket

Download an object

$ aws s3api --profile lc-objstore --endpoint-url https://sgusrgw1.llnl.gov:8082 get-object --bucket mybucket --key my-hosts downloaded-file

$ tail -1 downloaded-file

172.19.3.171  oslic21-san0 oslic21.llnl.gov oslic21

Delete an object

$ aws s3api --profile lc-objstore --endpoint-url https://sgusrgw1.llnl.gov:8082 delete-object --bucket mybucket --key my-hosts

S3 with python boto3

https://boto3.amazonaws.com/v1/documentation/api/latest/index.html

Create a virtual environment and install boto3

python3 -m venv boto-venv
source boto-venv/bin/activate
pip install -U pip
pip install boto3

Start python, import boto3 and setup the connection details

$ python
>>> import boto3
>>> import boto3.session
>>> session = boto3.session.Session(profile_name='lc-objstore')
>>> endpoint = 'https://sgusrgw1.llnl.gov:8082'
>>> s3 = session.resource(service_name='s3', endpoint_url=endpoint)
>>> client = s3.meta.client

List buckets

>>> for bucket in s3.buckets.all():
...   print(bucket.name)
...
mybucket

List files in a bucket

>>> for o in s3.Bucket('mybucket').objects.all():
...   print("Key: " + o.key)
...   print("Size: " + str(o.size))
...
...
Key: my-hosts
Size: 40246