S3
An S3 resource is characterised by S3 access.
Properties of an S3 resource
- Storage space: Storage space for S3 resources must always be requested via Coscine-JARDS.
- Use cases: large data sets, automated data submission, metadata management using Readme files, use of S3 storage space in conjunction with Jupyter Notebooks
- Access methods: User interface, API, S3
- Metadata: The entry of metadata via the metadata profile is not mandatory. The storage space application in Coscine-JARDS must therefore clearly explain what research data is involved and how it will be ensured that this data is described using metadata and made discoverable.
- Other special features: Users who wish to carry out complex S3 operations will find an overview of the S3 policies used in Coscine here.
Creating and using an S3 resource
Once the storage space request has successfully passed the review process and the quota has been allocated to the relevant project, you can create S3 resources within that project. Click the menu icon to open the Resource Configuration tab. There you will find the credentials and connection parameters you need to connect directly to the resource via the S3 protocol using an S3 client.
Warning: Recommended S3 clients
The Coscine team is unable to provide comprehensive support for the use of S3 clients. The S3 clients WinSCP, Cyberduck and MinIO Client have already been successfully tested with Coscine. You can find more detailed information on how to use them on their respective documentation pages. The Coscine team has no experience with the functionality of other applications (e.g. rclone, S3CMD).
Overview of credentials and connection parameters
To connect an S3 client to an S3 resource, you need various pieces of information, which may be named differently depending on the client. Generally speaking, a distinction is made between:
- Credentials: Credentials are similar to login or access details. They ensure that unauthorised third parties cannot access the data in the S3 resource. They also contain various permissions: Read Access Keys grant read-only access, whilst Write Access Keys grant both read and write access.
- Connection parameters: Connection parameters are details about the storage and the S3 resource within it (in the case of Coscine, the DataStorage.nrw), enabling the relevant S3 client to establish a connection.
The following tables provide an overview of the most common terms and values associated with this information:
| Name | Value | Note |
|---|---|---|
| Access Key (ID) | read_XXXXXX... or write_XXXXXX | The Read Access Key and Write Access Key each only work with the corresponding Secret Key. |
| Secret (Access) Key | XXXXXXXX-XXXX-XXXX-XXXX-... |
| Name | Value | Note |
|---|---|---|
| Server or Endpoint | global.datastorage.nrw | |
| Port | 443 | |
| Bucket Name | XXXXXXXX-XXXX-XXXX-... | |
| Path | Like Bucketname | Depending on the tool, the path can be found in the advanced or optional settings. However, it is essential for using S3 storage! |
Hint: Use the virtual host style
When connecting to DataStorage.nrw’s S3 resources, it is recommended that you specify the hostname using the Virtual Host style. Using the Path style may result in connection problems.