Ensure rc/stdout/stderr upload when task outputs missing when using AWS Batch #7826
+11
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixed an issue in the AWS Batch backend where jobs that fail before writing all their expected outputs would cause the delocalization step to fail entirely. This prevented the upload of crucial debugging files (return code, stdout, stderr) to S3.
When a task command fails (e.g., exit 1) before creating all declared outputs, the subsequent delocalization step would attempt to upload these missing files. With set -e enabled, the first failed upload would abort the entire delocalization block, meaning rc/stdout/stderr were never uploaded if they came after a missing output file.
This PR:
Example task to reproduce the issue:
Prior to this PR, the generated script would attempt to delocalize
goodbye.txtto S3, fail as it does not exist and exit before it uploaded the RC, stdout and stderr.Release Notes Confirmation
CHANGELOG.mdCHANGELOG.mdin this PRCHANGELOG.mdbecause it doesn't impact community usersTerra Release Notes