Quickstart (EN)

This guide contains a description of the basic steps necessary to start using the object storage service.

It assumes that the user’s PLGrid group has an active grant with resources in “ACK Cyfronet Object Storage” service and the user has an active “Access to Cyfronet Object Storage” service.

You can apply for a grant at https://portal.plgrid.pl/grants

You can request service access at https://portal.plgrid.pl/services/105

To use the service, active affiliation is also required. The list of user affiliations can be checked at https://portal.plgrid.pl/users/affiliations

Acquiring access credentials

Authentication with the service is performed using a pair of keys: access_key and secret_key. Example:

access_key: "plgs3p-plggbazaar-plguser1"
secret_key: "BCJuoRCv65CZxyIpseErB44RdXXDCH9Mg0fLn6SQ"

To generate them you have to sign in with your PLGrid account in the service management panel available at https://storage-panel.cloud.cyfronet.pl.

After you sign in, go to the “Credentials” page, choose your PLGrid group and storage region and press the “Generate credential” button.

You will then be shown the values of endpoint, access_key, and secret_key necessary to configure your tools.

The value of secret_key must be kept secret and not shared with other people.

The value of the endpoint depends on the region:

  • DC-Nawojki: s3.cloud.cyfronet.pl
  • DC-Podole: s3p.cloud.cyfronet.pl

Good to know that:

  • each member of the group has the same access to all of the group’s data
  • each combination of group-region-user corresponds to a different pair of (access_key, secret_key)
  • regenerating new access credentials for a combination of group-region-user invalidates the previous ones

Testing access credentials

You can test the access credentials using the web interface available at https://storagegw.cloud.cyfronet.pl

After choosing the region and entering valid access credentials you will see a list of buckets owned by the group.

The web interface can be used to create new buckets, browse a list of stored objects and perform on them basic operations (upload, download, delete)

Good to know that:

  • bucket names should contain only lowercase letters a-z, digits 0-9 and -
  • bucket names must be globally unique
    • we recommend using a unique group prefix, for example, bucket supergrp-test instead of test
    • trying to create a bucket with an already taken name shows the BucketAlreadyExists error

Configuring MinIO Client

Recommended tool for performing basic operations (creating buckets, uploading/downloading data, listing and deleting objects) is MinIO Client.

Warning: there might be issues with uploading files bigger than 80GB using this tool

When working on a computing cluster MinIo Client is available in the minio-client module.

In Linux systems the easiest way of getting started is to download the mc binary and place it in one of the directories in the PATH environmental variable:

$ curl --create-dirs -o ~/.local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
$ chmod +x ~/.local/bin/mc

The installation instructions for macOS and Windows are described in the official documentation: https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc

Warning: mc command might collide with the Midnight Commander tool.

Next, you have to create a new alias corresponding to a group’s pool of data on a region using the command mc alias set <alias-name> https://<endpoint>

Example:

$ mc alias set cyf-s3p https://s3p.cloud.cyfronet.pl
Enter Access Key: plgs3p-plggrp1-plguser1
Enter Secret Key: BCJuoRCv65CZxyIpseErB44RdXXDCH9Mg0fLn6SQ
Added `cyf-s3p` successfully.

MinIO Client - basic operations

Creating a new bucket:

$ mc mb <alias-name>/<bucket-name>

$ mc mb cyf-s3p/bucketA
Bucket created successfully `cyf-s3p/bucket-a`.

Listing buckets:

$ mc ls <alias-name>

$ mc ls cyf-s3p
[2022-09-27 14:40:33 CEST]     0B bucket-a/
[2022-10-06 12:34:28 CEST]     0B projxyz-data-2022/

Listing objects in a bucket:

$ mc ls <alias-name>/<bucket-name>/<object-name-prefix>

$ mc ls cyf-s3p/projxyz-data-2022
[2022-10-05 08:50:54 CEST] 536MiB STANDARD AlmaLinux-8-GenericCloud-8.5-20211119.x86_64.qcow2
[2022-09-27 14:40:54 CEST]  94MiB STANDARD traefik
[2022-10-06 15:37:35 CEST]     0B information/
[2022-10-06 15:37:35 CEST]     0B keygenme-py/
[2022-10-06 15:37:35 CEST]     0B pythonwrangling/
[2022-10-06 15:37:35 CEST]     0B waveaflag/

$ mc ls cyf-s3p/projxyz-data-2022/information
[2022-10-06 15:37:15 CEST] 858KiB STANDARD cat.jpg

Copying local file file1 to bucket bucket-a:

$ mc cp file1 cyf-s3p/bucket-a/

Copying directory data to bucket projxyz-data-2022:

$ mc cp --recursive data cyf-s3p/projxyz-data-2022

Copying objects with prefix data-w/ from bucket projxyz-data-2022 to directory data:

$ mc cp --recursive cyf-s3p/projxyz-data-2022/data-w/ data/

Removing a single object:

$ mc rm cyf-s3p/projxyz-data-2022/dane-w/plik1

Further information