Support per-bucket S3 configuration

[jorgee]

This pull request introduces support for per-bucket configuration in AWS S3 operations in Nextflow, allowing fine-grained control over settings like region, endpoint, and transfer options for each bucket. The changes refactor the codebase to leverage these per-bucket settings throughout S3 client creation and the .command.run generation, improving flexibility and correctness when working with multiple buckets.

A new buckets option has been added in the aws configuration to define bucket-specific configuration. Current client options are still valid and it is used for general bucket configuration. It will be applied to all the buckets that do not contain an specific configuration.

A specific bucket property is defined as follows:

aws.buckets.'ngi-igenomes'.anonymous = true

Specific configurations will override or complement general ones. So, if a property is defined in client and not in buckets.<bucket.name>, the bucket will be configured with the client value. If a property is defined in both or just in buckets.<bucket.name>, the bucket will be configured with the value in buckets.<bucket.name>.

Per-bucket configuration support

• Added new AwsBucketConfig model to represent S3 bucket-specific settings (region, endpoint, options etc.), and updated AwsConfig to parse and store a map of bucket configurations. (plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsBucketConfig.groovy, plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy) [1] [2] [3]
• Refactored S3 client creation in AwsClientFactory to accept a bucket name and use the corresponding bucket-specific configuration for credentials, endpoint, and path style access. (plugins/nf-amazon/src/main/nextflow/cloud/aws/AwsClientFactory.groovy) [1] [2] [3]

Task .command.run generation changes

• Updated batch executor (AwsBatchExecutor) and file copy strategy (AwsBatchFileCopyStrategy) to inject per-bucket CLI arguments (storage class, encryption, requester pays, etc.) into S3 upload/download commands, using the new configuration model. (plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchExecutor.groovy, plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchFileCopyStrategy.groovy) [1] [2] [3] [4] [5]

• Removed redundant getters for S3 transfer options from AwsOptions and replaced them with methods that generate CLI arguments based on bucket-specific configuration. (plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsOptions.groovy) [1] [2]

Configuration model refactoring

• Renamed AwsS3Config to AwsS3ClientConfig for clarity and updated references throughout the codebase. (plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy) [1] [2]

• Refactored region resolution logic to prioritize bucket-specific endpoint then global client endpoint, bucket region, aws.region, and finally default region. (plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy)

• Improved region resolution to check the AWS_REGION environment variable before AWS_DEFAULT_REGION, ensuring correct region selection in more scenarios. (plugins/nf-amazon/src/main/nextflow/cloud/aws/config/AwsConfig.groovy)

TODO

• Per bucket credentials: Easy to do in client, difficult to have a secure way to pass them to .command.run
• Extend to other cloud storage (Azure, Google)
• Fusion integration

close #4732
replaces #6553

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	cbfc9fd
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/691ef58772c3bf000836ca44