You can use one or multiple buckets on S3 to access data directly on your machine learning experiments and jobs.

Create an S3 bucket

You should create an S3 bucket (e.g. plx-storage).

You need to expose information about how to connect to the blob storage, the standard way is to expose these keys:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

And optionally these keys:

  • AWS_ENDPOINT_URL
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SECURITY_TOKEN
  • AWS_REGION

Create a secret or a config map for storing these keys

We recommend using a secret to store your access information json object:

kubectl create secret -n polyaxon generic s3-secret --from-literal=AWS_ACCESS_KEY_ID=key-id --from-literal=AWS_SECRET_ACCESS_KEY=hash-key

Use the secret name and secret key in your data persistence definition

connections:
- name: s3-dataset1
  kind: s3
  schema:
    bucket: "s3://bucket/"
  secret:
    name: "s3-secret"

If you want ot access multiple datasets using the same secret:

connections:
- name: s3-dataset1
  kind: s3
  schema:
    bucket: "s3://bucket/path1"
  secret:
    name: "s3-secret"
- name: s3-dataset1
  kind: s3
  schema:
    bucket: "s3://bucket/path2"
  secret:
    name: "s3-secret"

Update/Install Polyaxon deployment

You can deploy/upgrade your Polyaxon CE or Polyaxon Agent deployment with access to data on S3.

Access to the dataset in your experiments/jobs

To expose the connection secret to one of the containers in your jobs or services:

run:
  kind: job
  connections: [s3-dataset1]

Or

run:
  kind: job
  connections: [s3-dataset1, azure-dataset1]

Use the initializer to load the dataset

To use the artifacts initializer to load the dataset

run:
  kind: job
  init:
   - artifacts: [dirs: [...], files: [...]]
     connection: "s3-dataset1"

Use Polyaxon to access the dataset

This is optional, you can use any language or logic to interacts with S3 buckets.

Polyaxon has some built-in logic that you can leverage if you want.

To use that logic:

pip install polyaxon[s3]

All possible functions to use:

from polyaxon.connections.aws.s3 import S3Service

store = S3Service(...)

store.delete()
store.ls()
store.upload_file()
store.upload_dir()
store.download_file()
store.download_dir()