Overview
Coscine offers a variety of different storage systems for resources. Object storage-based systems such as the S3 storage store all data in so-called buckets. To uniquely identify data, the bucket name, the object key (name of the object), and the endpoint of the web service are required. Access to data in an S3 bucket is bound to access rights. To access buckets in S3 storage, so-called access keys must be used, which can be associated with either read or write permissions.
S3 resources, like all other resource types in Coscine, can be interacted with by using the REST API. Thus, buckets and objects can be created or deleted without having to use the Coscine web interface. The unique feature with S3 resources is that they can be interacted with entirely without Coscine. If access keys and bucket ID are known, S3 libraries such as boto3 or the S3 applications presented below can be used to interact directly with the data. In some cases, such as uploading and downloading very large files, this can be advantageous, but in doing so, metadata must be omitted due to bypassing the use of the web interface. Therefore, even with storage requests for an S3 resource, care is taken to ensure that good metadata management is considered and followed.
What exactly are S3 clients then?
S3 clients provide a direct connection to the S3 storage. They act completely independently of Coscine and therefore provide access to your data even if the Coscine web interface fails. For example, files can be uploaded or downloaded via S3 clients, but also edited accordingly. Since the S3 clients "communicate" directly with the underlying storage system, a faster upload and download of (larger) files is usually possible. However, for smaller files ( < 5Gb ) this difference hardly matters.
In the following documentation, we present the S3 applications Cyberduck, MinIO and WinSCP in more detail, with which an uncomplicated connection to the Coscine S3 storage is possible.
What does an example of interaction with the S3 storage using Python look like?
An example of the interaction with the S3 storage with Python is summarized in the following GitLab repository and can be used as inspiration for creating your own S3 clients.