MinIO

Here you will find an example of how to configure the MinIO Client (mc), a free S3 client for Linux, macOS and Windows.

MinIO client - S3 via commandline

The open source software MinIO differs from other clients like Cyberduck and WinSCP, as it does not offer a user interface, but only the command line interface (cli) Therefore minIO is more commonly used by users with higher technical knowledge who can teach themselves via documentation and help from online communities.

Download of the MinIO client

In the GitHub-Repository of the MinIO client, you find multiple download options. For example binary files for Linux, maxOS or Windows. Please note that the client must be installed on the system from or to which you wish to migrate data (e.g., installation on a computing cluster as well).

S3-Connections via command line (Linux & Windows)

Using the client in the command line is done by starting with the command mc which represents the client. Afterwards you might add typical commands like the linux commands cp, ls, cat, head, mv, and rm for S3 compatible storage services.

Configuration file setup

While it's possible to manually enter all parameters via the command line, using a configuration file is recommended. Using such a file (config.json) offers advantages in terms of consistency, reproducibility, and security. The configuration file is created by setting an alias and can then be manually modified if necessary. The command to create the configuration file is

mc alias set <my_resource_alias> https://global.datastorage.nrw:443/

The name for the placeholder term my_resource_alias can be freely chosen. The resulting configuration file can be found under ~/.mc/config.json under Linux or ?%USERPROFILE%\.mc\config.json under Windows. It should look like the following code:

{
    "version": "10",
    "aliases": {
        "my_resource_alias": {
        "url": "https://global.datastorage.nrw:443/",
        "accessKey": "<READ_OR_WRITE_ACCESS_KEY_FROM_COSCINE>",
        "secretKey": "<READ_OR_WRITE_SECRET_KEY_FROM_COSCINE>",
        "api": "s3v4",
        "path": "off"
        }
    }
} 

The access key and secret key for input into the configuration file, can be found in the resource configuration in Coscine. Depending on the usecase, you can add the key pair for either read or write permissions.

File operations:

After the creation of the configuration file, you can use different commands with the client mc. Typical operations are listing, uploading files and downloading files. For more information please visit the corresponding Git Repository.

Warning

For access to the Datastorage.nrw the virtual host style needs to be used, to satisfy the demands of the manufacturer. Additionally the option "path" needs to be set to "off".

Keep in mind that overwriting an existing file will create a new file or object version. File versions will block Storage space until the end of the retention period.

List files in a resource

Run

mc ls my_resource_alias/<bucket name>

Upload a file into a resource

Run

mc cp ./localFile.csv my_resource_alias/<bucket name>/

Download a file from a resource

Run

mc cp my_resource_alias/<bucket name>/data.csv ./