fix(taskfiles): Output unexpected errors in checksum tasks; Call gtar on macos; Add unit tests for checksum tasks (fixes #67). #81
Conversation
WalkthroughAdds macOS prerequisites to README. Refactors checksum tasks to use an OS-aware streaming tar → md5sum pipeline, introduces TMP_ERR_LOG, EXCLUDE_PATTERNS, and FAIL-controlled fail-soft/fail-fast behavior; validate now computes to a temp file and conditionally fails or cleans up. Adds checksum test suite and integrates it into tests aggregation. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Caller
participant Compute as checksum:compute
participant TAR as tar/gtar
participant MD5 as md5sum
participant FS as Filesystem
participant ERR as TMP_ERR_LOG
Caller->>Compute: invoke(INCLUDE_PATTERNS, EXCLUDE_PATTERNS, CHECKSUM_FILE, FAIL)
Compute->>Compute: select TAR (darwin → gtar, else tar)
Compute->>TAR: tar create stdout (include/exclude)
TAR-->>ERR: stderr → TMP_ERR_LOG
TAR-->>MD5: stdout (archive stream)
MD5-->>FS: write CHECKSUM_FILE
alt tar or md5 failure
Compute->>ERR: read errors
alt FAIL=true
Compute-->>Caller: exit 1 with error details
else FAIL=false
Compute-->>Caller: exit 0 (non-fatal)
end
else success
Compute-->>Caller: exit 0
end
sequenceDiagram
autonumber
participant Caller
participant Validate as checksum:validate
participant Compute as checksum:compute(tmp)
participant FS as Filesystem
participant ERR as TMP_ERR_LOG
Caller->>Validate: invoke(INCLUDE_PATTERNS, EXCLUDE_PATTERNS, CHECKSUM_FILE, FAIL)
Validate->>FS: verify INCLUDE paths exist
FS-->>Validate: missing paths? (log to TMP_ERR_LOG)
Validate->>Compute: compute TMP_CHECKSUM_FILE
Compute-->>Validate: exit code
Validate->>FS: cmp TMP_CHECKSUM_FILE vs CHECKSUM_FILE
alt any failure or mismatch
alt FAIL=true
Validate->>ERR: aggregate logs
Validate-->>Caller: exit 1 with error details
else FAIL=false
Validate->>FS: remove CHECKSUM_FILE (allow rerun)
Validate-->>Caller: exit 0
end
else success
Validate-->>Caller: exit 0
end
Validate->>FS: cleanup TMP files (defer)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: ASSERTIVE Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (2)📚 Learning: 2025-08-26T16:37:16.741ZApplied to files:
📚 Learning: 2025-08-26T16:37:16.741ZApplied to files:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
taskfiles/checksum/tests.yaml (1)
141-149: Status inputs: consider using directory contents as sources for better cache invalidationCurrently
create-dir-with-checksumusessources: ["{{.TASKFILE}}"], so changes to files under DIR won’t trigger a rebuild via Task’s up-to-date checks; you rely on checksum:validate instead. That’s fine for this test, but if you want Task’s native skipping to reflect content changes, consider adding the directory as a source (globbed).Optional example:
- sources: ["{{.TASKFILE}}"] + sources: + - "{{.DIR}}/**" + - "{{.TASKFILE}}"
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (4)
README.md(2 hunks)exports/taskfiles/utils/checksum.yaml(3 hunks)taskfiles/checksum/tests.yaml(1 hunks)taskfiles/tests.yaml(1 hunks)
🧰 Additional context used
🪛 LanguageTool
README.md
[grammar] ~17-~17: The operating system from Apple is written “macOS”.
Context: ...es that are not always pre-installed on MacOS. You may need to install the following ...
(MAC_OS)
🔇 Additional comments (2)
taskfiles/tests.yaml (1)
13-16: LGTM: checksum tests are now part of the default suiteIncluding checksum tests in tasks.all makes the new behaviour exercised continuously. Good move, and the commented boost line keeps context.
taskfiles/checksum/tests.yaml (1)
64-64: Avoid Bash-only process substitution and make mtime checks cross-platform
cmp -s '…' <(date -r …)requires Bash and will fail under /bin/sh (dash) and on macOS (date -r semantics differ).- Use stat with an OS guard and plain files for comparison.
Apply:
- - "date -r '{{.CHECKSUM_FILE}}' > '{{.CHECKSUM_MOD_TS}}'" + - |- + if [ "$(uname)" = "Darwin" ]; then + stat -f %m '{{.CHECKSUM_FILE}}' > '{{.CHECKSUM_MOD_TS}}' + else + stat -c %Y '{{.CHECKSUM_FILE}}' > '{{.CHECKSUM_MOD_TS}}' + fi @@ - - "cmp -s '{{.CHECKSUM_MOD_TS}}' <(date -r '{{.CHECKSUM_FILE}}')" + - |- + if [ "$(uname)" = "Darwin" ]; then + stat -f %m '{{.CHECKSUM_FILE}}' > '{{.CHECKSUM_FILE}}-mod-ts.new.txt' + else + stat -c %Y '{{.CHECKSUM_FILE}}' > '{{.CHECKSUM_FILE}}-mod-ts.new.txt' + fi + cmp -s '{{.CHECKSUM_MOD_TS}}' '{{.CHECKSUM_FILE}}-mod-ts.new.txt'Also applies to: 73-73
⛔ Skipped due to learnings
Learnt from: davidlion PR: y-scope/yscope-dev-utils#70 File: taskfiles/remote/tests.yaml:62-72 Timestamp: 2025-07-12T05:52:11.439Z Learning: The `date -r <file>` command is supported on both Linux and macOS to get file modification times, making it portable across these platforms.Learnt from: AVMatthews PR: y-scope/yscope-dev-utils#13 File: taskfiles/utils.yml:149-149 Timestamp: 2024-10-28T19:02:02.131Z Learning: In `taskfiles/utils.yml`, avoid using GNU-specific options like `-r` and `--no-run-if-empty` with `xargs` for better portability; instead, use `find`'s `-exec` option.
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
exports/taskfiles/utils/checksum.yaml (1)
30-37: Globbing with spaces may mis-split; verify this is acceptablePatterns are intentionally unquoted so the shell expands them, but this will split on whitespace in file names. If your repo allows spaces, you’ll need a different enumeration strategy (e.g., tar -T with a nul list). If spaces are out of scope, consider documenting that.
Also applies to: 55-56
♻️ Duplicate comments (5)
README.md (1)
15-22: macOS prerequisites: add PATH instructions for Homebrew “gnubin” and clarify tool namesReaders will otherwise miss that md5sum and gtar won’t exist unless gnubin is prepended to PATH on macOS. Also, prefer “Homebrew packages” and remove the escaped colons; use brief “why” notes.
-### macOS - -The exported tasks use GNU utilities that are not always pre-installed on macOS. You may need to -install the following brew packages and add their executables to your PATH: - -* [coreutils]\: `md5sum` -* [gnu-tar]\: `gtar` +### macOS + +The exported tasks use GNU utilities that are not always pre-installed on macOS. Install the +following Homebrew packages and add their “gnubin” shims to your PATH so `md5sum` and `gtar` +are available under the expected names: + +* [coreutils] (provides `md5sum` and friends) +* [gnu-tar] (provides `gtar`) + +Add to your shell profile (adjust for Intel vs Apple Silicon): + +```sh +export PATH="$(brew --prefix coreutils)/libexec/gnubin:$(brew --prefix gnu-tar)/libexec/gnubin:$PATH" +```exports/taskfiles/utils/checksum.yaml (4)
19-21: Nil-safe handling of EXCLUDE_PATTERNSRelying on
ref: "default (list) .EXCLUDE_PATTERNS"is brittle; use a nil-safe default at the callsite so ranging a missing var doesn’t error.- EXCLUDE_PATTERNS: - ref: "default (list) .EXCLUDE_PATTERNS" + EXCLUDE_PATTERNS: + ref: ".EXCLUDE_PATTERNS" @@ - {{- range .EXCLUDE_PATTERNS}} + {{- range (default (list) .EXCLUDE_PATTERNS) }} --exclude="{{.}}" \ {{- end}}If you prefer keeping the default at definition time, consider a shell-evaluated var:
+ EXCLUDE_PATTERNS: + sh: 'echo ""' # default to empty list; still wrap with default(list) when rangingAlso applies to: 51-53
38-66: Pipeline can produce bogus checksums if tar fails; avoid relying on pipeline exit statusBecause the pipeline status is that of
md5sum(no pipefail), tar errors won’t cause the if-guard to fail, and you may hash empty/partial input. Write tar output to a temp file, check tar’s exit, then hash and atomically move the checksum.- - defer: "rm -f '{{.TMP_ERR_LOG}}'" + - defer: "rm -f '{{.TMP_ERR_LOG}}' '{{.TMP_TAR_FILE}}' '{{.TMP_CHECKSUM_FILE}}'" - |- - if ! \ - {{.TAR}} \ - --create \ - --file - \ + if ! \ + {{.TAR}} \ + --create \ + --file "{{.TMP_TAR_FILE}}" \ --group 0 \ --mtime "UTC 1970-01-01" \ --numeric-owner \ --owner 0 \ --sort name \ --no-anchored \ --wildcards \ - {{- range .EXCLUDE_PATTERNS}} + {{- range (default (list) .EXCLUDE_PATTERNS) }} --exclude="{{.}}" \ {{- end}} {{- range .INCLUDE_PATTERNS}} {{.}} \ {{- end}} - 2> "{{.TMP_ERR_LOG}}" \ - | md5sum > "{{.CHECKSUM_FILE}}" \ + 2> "{{.TMP_ERR_LOG}}" \ ; then - rm "{{.CHECKSUM_FILE}}" + rm -f "{{.CHECKSUM_FILE}}" {{- if eq "true" .FAIL}} - printf "[{{.TASK}} error] failed with:\n%s\n" "$(cat {{.TMP_ERR_LOG}})" + printf "[{{.TASK}} error] tar failed with:\n%s\n" "$(cat "{{.TMP_ERR_LOG}}")" exit 1 {{- else}} exit 0 {{- end}} fi + if ! {{.MD5_CMD}} "{{.TMP_TAR_FILE}}" > "{{.TMP_CHECKSUM_FILE}}"; then + printf "[{{.TASK}} error] failed to compute checksum for %s\n" "{{.TMP_TAR_FILE}}" + exit 1 + fi + mv -f "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}"Add the supporting vars:
vars: @@ TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" + TMP_TAR_FILE: "{{.CHECKSUM_FILE}}.tar.tmp" + TMP_CHECKSUM_FILE: "{{.CHECKSUM_FILE}}.tmp"
23-24: Make the checksum tool portable via MD5_CMD and use it in commandsOn macOS,
md5sumoften exists only via coreutils gnubin or asgmd5sum; falling back tomd5 -rimproves portability.vars: @@ - TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" + TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" + MD5_CMD: + sh: | + if command -v md5sum >/dev/null 2>&1; then echo md5sum; + elif command -v gmd5sum >/dev/null 2>&1; then echo gmd5sum; + elif command -v md5 >/dev/null 2>&1; then echo 'md5 -r'; + else echo 'false'; fi @@ - | md5sum > "{{.CHECKSUM_FILE}}" \ + | {{.MD5_CMD}} > "{{.CHECKSUM_FILE}}" \Also applies to: 58-59
84-87: Validation: accumulate errors and append logs; ensure any failure flips the subshell statusAt present, each missing include overwrites the log and the loop doesn’t guarantee a final non-zero when earlier iterations failed; also use append and printf for multi-line safety.
- - defer: |- - rm -f "{{.TMP_CHECKSUM_FILE}}" - rm -f "{{.TMP_ERR_LOG}}" + - defer: |- + rm -f "{{.TMP_CHECKSUM_FILE}}" "{{.TMP_ERR_LOG}}" - |- ( - {{- range .INCLUDE_PATTERNS}} - for path in {{.}}; do - test -e "${path}" \ - || ( - echo "Include path does not exist: ${path}" > "{{$.TMP_ERR_LOG}}" - exit 1 - ) - done && \ - {{- end}} - ( - cmp -s "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}" \ - || ( - echo "cmp failed for '{{.TMP_CHECKSUM_FILE}}' '{{.CHECKSUM_FILE}}'" \ - > "{{.TMP_ERR_LOG}}" - exit 1 - ) - ) + fail=0 + : > "{{.TMP_ERR_LOG}}" + {{- range .INCLUDE_PATTERNS}} + for path in {{.}}; do + if [ ! -e "$path" ]; then + printf 'Include path does not exist: %s\n' "$path" >> "{{$.TMP_ERR_LOG}}" + fail=1 + fi + done + {{- end}} + if ! cmp -s "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}"; then + printf "cmp failed for '%s' '%s'\n" "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}" >> "{{.TMP_ERR_LOG}}" + fail=1 + fi + test $fail -eq 0 ) \ || ( {{- if eq "true" .FAIL}} - printf "[{{.TASK}} error] failed with:\n%s\n" "$(cat {{.TMP_ERR_LOG}})" + printf "[{{.TASK}} error] failed with:\n%s\n" "$(cat "{{.TMP_ERR_LOG}}")" exit 1 {{- else}} rm -f "{{.CHECKSUM_FILE}}" {{- end}} )Also applies to: 100-130
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
README.md(2 hunks)exports/taskfiles/utils/checksum.yaml(2 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-08-26T16:37:16.718Z
Learnt from: davidlion
PR: y-scope/yscope-dev-utils#81
File: exports/taskfiles/utils/checksum.yaml:16-20
Timestamp: 2025-08-26T16:37:16.718Z
Learning: In Task (Taskfile), `OS` is a built-in function, not a variable. The correct syntax is `eq OS "darwin"` without a dot prefix. Using `.OS` would be incorrect syntax.
Applied to files:
exports/taskfiles/utils/checksum.yaml
📚 Learning: 2025-08-26T16:37:16.718Z
Learnt from: davidlion
PR: y-scope/yscope-dev-utils#81
File: exports/taskfiles/utils/checksum.yaml:16-20
Timestamp: 2025-08-26T16:37:16.718Z
Learning: Task's templating system has built-in functions like `OS` that don't require the dot prefix, unlike user-defined variables which do require the dot prefix (e.g., `.VARIABLE_NAME`).
Applied to files:
exports/taskfiles/utils/checksum.yaml
🔇 Additional comments (2)
README.md (1)
53-54: Link references LGTMThe reference-style links for coreutils and gnu-tar are correct and helpful.
exports/taskfiles/utils/checksum.yaml (1)
23-24: Correct OS check usage—good catchUsing the Task built-in function
OSwithout a dot is correct here and selects GNU tar on macOS as intended.
| TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" | ||
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Prefer capability detection over OS check for TAR
Some Linux/Alpine images ship BusyBox tar which lacks these flags. Consider preferring gtar if present, falling back to tar.
- TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}"
+ TAR:
+ sh: |
+ if command -v gtar >/dev/null 2>&1; then echo gtar; else echo tar; fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" | |
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" | |
| TAR: | |
| sh: | | |
| if command -v gtar >/dev/null 2>&1; then echo gtar; else echo tar; fi | |
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" |
🤖 Prompt for AI Agents
In exports/taskfiles/utils/checksum.yaml around lines 23-24, replace the
OS-based TAR selection with a runtime capability check that prefers gtar when
present and falls back to tar; specifically, change the template expression to
run a shell command like `command -v gtar` (or `which gtar`) at runtime and use
gtar if it exists, otherwise use tar, keeping TMP_ERR_LOG unchanged.
There was a problem hiding this comment.
we will improve this once we add Linux/Alphine CIs
There was a problem hiding this comment.
@Bill-hbrhbr, understood! That makes sense to defer this until you have Linux/Alpine CI environments to properly test the capability detection. Would you like me to create a GitHub issue to track this improvement for when the CI infrastructure is ready?
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (3)
exports/taskfiles/utils/checksum.yaml (3)
23-24: OS function usage is correct; prefer capability detection for broader portabilityConfirmed per our prior learning: OS is a built-in function (no dot), so your current expression is valid on Task. To harden across BusyBox/non-GNU environments that nonetheless ship gtar, detect gtar at runtime and fall back to tar.
Apply:
- TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" + TAR: + sh: | + if command -v gtar >/dev/null 2>&1; then + echo gtar + else + echo tar + fi
104-122: Validation logic: existence checks don’t affect overall status; error log gets overwritten
- test -e … || ( …; exit 1 ) happens in a subshell; the for-loop continues and the outer group’s status often reflects only the last iteration.
- Using > overwrites the log; earlier errors are lost.
Accumulate failures, append errors, and assert at the end. Also append cmp failures.
Apply:
- ( - {{- range .INCLUDE_PATTERNS}} - for path in {{.}}; do - test -e "${path}" \ - || ( - echo "Include path does not exist: ${path}" > "{{$.TMP_ERR_LOG}}" - exit 1 - ) - done && \ - {{- end}} - ( - cmp -s "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}" \ - || ( - echo "cmp failed for '{{.TMP_CHECKSUM_FILE}}' '{{.CHECKSUM_FILE}}'" \ - > "{{.TMP_ERR_LOG}}" - exit 1 - ) - ) - ) \ + ( + fail=0 + : > "{{.TMP_ERR_LOG}}" + {{- range .INCLUDE_PATTERNS}} + for path in {{.}}; do + if [ ! -e "${path}" ]; then + echo "Include path does not exist: ${path}" >> "{{$.TMP_ERR_LOG}}" + fail=1 + fi + done + {{- end}} + if ! cmp -s "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}"; then + echo "cmp failed for '{{.TMP_CHECKSUM_FILE}}' '{{.CHECKSUM_FILE}}'" >> "{{.TMP_ERR_LOG}}" + fail=1 + fi + test $fail -eq 0 + ) \
51-56: Nil-safety for EXCLUDE_PATTERNS during rangeEven if you fix vars, being defensive at the use site avoids surprises when this task is embedded under different callers.
Apply:
- {{- range .EXCLUDE_PATTERNS}} + {{- range (default (list) .EXCLUDE_PATTERNS) }}
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
exports/taskfiles/utils/checksum.yaml(2 hunks)
🧰 Additional context used
🧠 Learnings (3)
📚 Learning: 2025-08-26T16:37:16.718Z
Learnt from: davidlion
PR: y-scope/yscope-dev-utils#81
File: exports/taskfiles/utils/checksum.yaml:16-20
Timestamp: 2025-08-26T16:37:16.718Z
Learning: In Task (Taskfile), `OS` is a built-in function, not a variable. The correct syntax is `eq OS "darwin"` without a dot prefix. Using `.OS` would be incorrect syntax.
Applied to files:
exports/taskfiles/utils/checksum.yaml
📚 Learning: 2025-08-26T16:37:16.718Z
Learnt from: davidlion
PR: y-scope/yscope-dev-utils#81
File: exports/taskfiles/utils/checksum.yaml:16-20
Timestamp: 2025-08-26T16:37:16.718Z
Learning: Task's templating system has built-in functions like `OS` that don't require the dot prefix, unlike user-defined variables which do require the dot prefix (e.g., `.VARIABLE_NAME`).
Applied to files:
exports/taskfiles/utils/checksum.yaml
📚 Learning: 2024-10-28T19:02:02.131Z
Learnt from: AVMatthews
PR: y-scope/yscope-dev-utils#13
File: taskfiles/utils.yml:149-149
Timestamp: 2024-10-28T19:02:02.131Z
Learning: In `taskfiles/utils.yml`, avoid using GNU-specific options like `-r` and `--no-run-if-empty` with `xargs` for better portability; instead, use `find`'s `-exec` option.
Applied to files:
exports/taskfiles/utils/checksum.yaml
🔇 Additional comments (1)
exports/taskfiles/utils/checksum.yaml (1)
19-21: Critical: invalid use of ref with template functions; EXCLUDE_PATTERNS won’t resolveTask’s ref cannot evaluate template functions. As written, ref: "default (list) .EXCLUDE_PATTERNS" will not work and may break the task. Use a templated string instead so default/list are evaluated.
Apply:
- EXCLUDE_PATTERNS: - ref: "default (list) .EXCLUDE_PATTERNS" + EXCLUDE_PATTERNS: "{{ default (list) .EXCLUDE_PATTERNS }}"Likely an incorrect or invalid review comment.
| TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" | ||
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" | ||
| requires: |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add MD5_CMD and temp artifact vars for portability and atomic writes
md5sum is not present by default on macOS; Homebrew installs gmd5sum. Also, writing directly to CHECKSUM_FILE risks partial files on failure. Introduce MD5_CMD and temp files now (used in compute refactor below).
Apply:
TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp"
+ TMP_TAR_FILE: "{{.CHECKSUM_FILE}}.tar.tmp"
+ TMP_CHECKSUM_FILE: "{{.CHECKSUM_FILE}}.tmp"
+ MD5_CMD:
+ sh: |
+ if command -v gmd5sum >/dev/null 2>&1; then
+ echo gmd5sum
+ elif command -v md5sum >/dev/null 2>&1; then
+ echo md5sum
+ elif command -v md5 >/dev/null 2>&1; then
+ # macOS fallback; -r prints digest first to match *md5sum output style
+ echo 'md5 -r'
+ else
+ echo 'false' # no-op that will fail
+ fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" | |
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" | |
| requires: | |
| TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" | |
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" | |
| TMP_TAR_FILE: "{{.CHECKSUM_FILE}}.tar.tmp" | |
| TMP_CHECKSUM_FILE: "{{.CHECKSUM_FILE}}.tmp" | |
| MD5_CMD: | |
| sh: | | |
| if command -v gmd5sum >/dev/null 2>&1; then | |
| echo gmd5sum | |
| elif command -v md5sum >/dev/null 2>&1; then | |
| echo md5sum | |
| elif command -v md5 >/dev/null 2>&1; then | |
| # macOS fallback; -r prints digest first to match *md5sum output style | |
| echo 'md5 -r' | |
| else | |
| echo 'false' # no-op that will fail | |
| fi | |
| requires: |
🤖 Prompt for AI Agents
In exports/taskfiles/utils/checksum.yaml around lines 23-25, add a
platform-aware MD5_CMD variable and explicit temp artifact vars for atomic
writes: define MD5_CMD to use gmd5sum on darwin and md5sum otherwise, and add
CHECKSUM_TMP (e.g. "{{.CHECKSUM_FILE}}.tmp") and retain TMP_ERR_LOG (or rename
to CHECKSUM_ERR_LOG if desired) so compute steps can write to temp files and
then atomically mv to CHECKSUM_FILE; update downstream steps to use these new
vars for generating checksums and error logs.
| - defer: "rm -f '{{.TMP_ERR_LOG}}'" | ||
| - |- | ||
| if ! \ | ||
| {{.TAR}} \ | ||
| --create \ | ||
| --file - \ | ||
| --group 0 \ | ||
| --mtime "UTC 1970-01-01" \ | ||
| --numeric-owner \ | ||
| --owner 0 \ | ||
| --sort name \ | ||
| --no-anchored \ | ||
| --wildcards \ | ||
| {{- range .EXCLUDE_PATTERNS}} | ||
| --exclude="{{.}}" \ | ||
| {{- end}} | ||
| {{- range .INCLUDE_PATTERNS}} | ||
| {{.}} \ | ||
| {{- end}} | ||
| 2> "{{.TMP_ERR_LOG}}" \ | ||
| | md5sum > "{{.CHECKSUM_FILE}}" \ | ||
| ; then |
There was a problem hiding this comment.
Tar|md5sum pipeline can silently succeed on tar errors; compute may emit bogus checksums
Without pipefail, the if condition observes md5sum’s status only; tar failures can be masked and an empty/partial stream hashed. Even with pipefail, md5sum can write a truncated checksum before failure is detected. Write tar output to a temporary file, check tar status explicitly, then hash the file into a temporary checksum and atomically move it into place. Also, consolidate cleanup in defer.
Apply:
- - defer: "rm -f '{{.TMP_ERR_LOG}}'"
+ - defer: "rm -f '{{.TMP_ERR_LOG}}' '{{.TMP_TAR_FILE}}' '{{.TMP_CHECKSUM_FILE}}'"
- |-
- if ! \
- {{.TAR}} \
- --create \
- --file - \
+ if ! \
+ {{.TAR}} \
+ --create \
+ --file "{{.TMP_TAR_FILE}}" \
--group 0 \
--mtime "UTC 1970-01-01" \
--numeric-owner \
--owner 0 \
--sort name \
--no-anchored \
--wildcards \
- {{- range .EXCLUDE_PATTERNS}}
+ {{- range (default (list) .EXCLUDE_PATTERNS) }}
--exclude="{{.}}" \
{{- end}}
{{- range .INCLUDE_PATTERNS}}
{{.}} \
{{- end}}
- 2> "{{.TMP_ERR_LOG}}" \
- | md5sum > "{{.CHECKSUM_FILE}}" \
+ 2> "{{.TMP_ERR_LOG}}" \
; then
rm -f "{{.CHECKSUM_FILE}}"
{{- if eq "true" .FAIL}}
printf "[{{.TASK}} error] failed with:\n%s\n" "$(cat {{.TMP_ERR_LOG}})"
exit 1
{{- else}}
exit 0
{{- end}}
fi
+ if ! {{.MD5_CMD}} "{{.TMP_TAR_FILE}}" > "{{.TMP_CHECKSUM_FILE}}" 2>>"{{.TMP_ERR_LOG}}"; then
+ rm -f "{{.CHECKSUM_FILE}}"
+ printf "[%s error] md5 failed; details:\n%s\n" "{{.TASK}}" "$(cat "{{.TMP_ERR_LOG}}")"
+ exit 1
+ fi
+ mv -f "{{.TMP_CHECKSUM_FILE}}" "{{.CHECKSUM_FILE}}"Note: Also harden the loop over EXCLUDE_PATTERNS by wrapping with default (list) to be nil-safe.
Also applies to: 61-67
🤖 Prompt for AI Agents
In exports/taskfiles/utils/checksum.yaml around lines 38-59 (and similarly for
61-67), the tar | md5sum pipeline can mask tar failures and produce bogus
checksums; change the logic to: write tar output to a temporary archive file,
check the tar command exit status explicitly (fail early if tar failed and log
stderr), then compute the md5sum of that temporary archive into a temporary
checksum file and atomically move the checksum into place only on success;
consolidate all temporary-file cleanup in the existing defer, and make the loop
over EXCLUDE_PATTERNS nil-safe by iterating over a default empty list when the
variable is unset.
💡 Verification agent
🧩 Analysis chain
Inconsistency with prior claims of “addressed” fixes
Past discussion indicated the tar→tmp+hash approach and the validate fail-accumulation were addressed. The current file still uses a pipe for compute and per-iteration exits/overwrites in validate. Treating this as still-open in this PR.
Run to confirm no other copies linger:
Also applies to: 104-122
🏁 Script executed:
#!/usr/bin/env bash
# Find unsupported uses of "ref:" with template functions and risky tar|md5sum pipelines
rg -nC2 -e $'ref:\\s*\"default\\s*\\(list\\)' -e $'\\|\\s*md5sum\\b' exports/taskfiles/utils/checksum.yamlLength of output: 438
The following will confirm whether any other instances of the legacy pipe-to-md5sum or template “ref:” patterns remain:
🏁 Script executed:
#!/usr/bin/env bash
# Search for any md5sum pipelines
rg -nC2 "md5sum" exports/taskfiles/utils/checksum.yaml
# Search for tar --create usage (to spot other direct streams)
rg -nC2 "tar --create" exports/taskfiles/utils/checksum.yaml
# Search for template ref: calls
rg -nC2 "ref:" exports/taskfiles/utils/checksum.yamlLength of output: 1170
Critical: Legacy tar | md5sum pipeline and unsupported template refs still present
Our grep verification confirms that the original issues remain in exports/taskfiles/utils/checksum.yaml:
- Line 20:
ref: "default (list) .EXCLUDE_PATTERNS"
(unsupported template “ref:” syntax) - Lines 96–98:
ref: ".INCLUDE_PATTERNS"
ref: ".EXCLUDE_PATTERNS"
(additional unsupported template “ref:” calls) - Line 58:
| md5sum > "{{.CHECKSUM_FILE}}"
(still using a direct tar→pipe→md5sum approach, rather than writing to a temp file and then hashing)
These occurrences indicate that neither the tar→tmp+hash refactor nor the validate fail-accumulation fix was applied in this PR. Please address the following before merging:
- Replace the inline tar | md5sum pipeline with the tmp‐file+hash approach discussed previously.
- Remove or rewrite all unsupported
ref:template calls (default (list)and bare".INCLUDE_PATTERNS"/".EXCLUDE_PATTERNS") to use the standard Taskfile templating functions.
🤖 Prompt for AI Agents
In exports/taskfiles/utils/checksum.yaml around lines 38 to 59 (and also remove
unsupported refs at lines ~20 and ~96-98), the current tar | md5sum pipeline and
unsupported template "ref:" usages remain; replace the inline pipe with the
temp-file + hash approach (have tar write to a temporary file, check for tar
errors, then compute the checksum from that temp file and atomically write
CHECKSUM_FILE), and eliminate all unsupported ref: template calls by using
standard Taskfile templating (use the built-in default function and range over
.EXCLUDE_PATTERNS and .INCLUDE_PATTERNS instead of ref: "default (list) ..." or
ref: ".INCLUDE_PATTERNS"/".EXCLUDE_PATTERNS"); ensure error output is captured
to TMP_ERR_LOG and cleanup/defer stays intact.
There was a problem hiding this comment.
isn't the checksum file deleted if there's any failure within the tar->pipe->md5sum process? I don't think we need to worry about truncated checksums.
Also ref: "default (list) .EXCLUDE_PATTERNS" is redundant. We already do this in the vars: section.
There was a problem hiding this comment.
🧩 Analysis chain
@Bill-hbrhbr, let me address both points:
1. Pipefail and truncated checksums
You're right that rm -f "{{.CHECKSUM_FILE}}" is in the error branch, but without set -o pipefail, that branch won't execute when tar fails. Here's why:
Without pipefail, the if ! condition only checks the last command in the pipeline (md5sum). If tar fails but md5sum succeeds (it will hash whatever partial data it receives, even an empty stream), the condition evaluates to false (because md5sum returned 0), so the error branch is skipped entirely. The bogus checksum file remains.
According to the AI summary, this PR removed the top-level set: ["u", "pipefail"]. Without pipefail, tar failures can indeed produce truncated checksums that slip through.
2. The ref: syntax
Let me verify whether ref: "default (list) .EXCLUDE_PATTERNS" is valid Taskfile syntax.
🏁 Script executed:
if ! tar ... | md5sum > checksum ; then
rm checksum # Only runs if md5sum fails
fi
Length of output: 82
🏁 Script executed:
#!/bin/bash
# Fetch Taskfile documentation on variables and ref syntax
curl -s "https://taskfile.dev/api/schema.json" | jq -r '.definitions.vars.properties' 2>/dev/null || echo "Schema not available via API"
# Check if there are any examples in the Taskfile docs
curl -sL "https://raw.githubusercontent.com/go-task/task/main/docs/docs/usage.md" | grep -A5 -B5 "ref:" || echo "No ref: examples found"
Length of output: 51
🏁 Script executed:
#!/bin/bash
# Search Taskfile documentation for ref syntax
curl -sL "https://taskfile.dev/usage/" 2>/dev/null | grep -A10 -B10 "ref" || echo "Usage page not accessible"
# Try to get the JSON schema definition
curl -sL "https://json.schemastore.org/taskfile.json" 2>/dev/null | jq '.definitions.var' 2>/dev/null || echo "Schema not accessible"
# Search for Taskfile variable examples
curl -sL "https://raw.githubusercontent.com/go-task/task/main/docs/docs/experiments/any-variables.md" 2>/dev/null || echo "Any-variables doc not found"
Length of output: 286751
@Bill-hbrhbr thanks—two quick points:
- Pipeline semantics: without pipefail, the if condition only sees md5sum’s status. So when tar fails but still writes nothing/partial to stdout, md5sum happily hashes whatever it got and returns 0; the error branch (which rm -f’s the checksum) won’t run, leaving a bogus checksum behind. In this PR we dropped the global pipefail (per docs), so the compute pipeline needs its own pipefail to make your rm -f path effective.
Minimal fix (adds pipefail just for compute):
version: "3"
tasks:
compute:
internal: true
label: "{{.TASK}}-{{.CHECKSUM_FILE}}"
silent: true
set: ["pipefail"] # ensure tar failures aren’t masked by md5sum
vars:
EXCLUDE_PATTERNS:
ref: "default (list) .EXCLUDE_PATTERNS"
FAIL: "{{if eq \"false\" .FAIL}}false{{else}}true{{end}}"
TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}"
TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp"
requires:
vars: ["CHECKSUM_FILE","INCLUDE_PATTERNS"]
cmds:
- defer: "rm -f '{{.TMP_ERR_LOG}}'"
- |-
if ! \
{{.TAR}} \
--create \
--file - \
--group 0 \
--mtime "UTC 1970-01-01" \
--numeric-owner \
--owner 0 \
--sort name \
--no-anchored \
--wildcards \
{{- range .EXCLUDE_PATTERNS}}
--exclude="{{.}}" \
{{- end}}
{{- range .INCLUDE_PATTERNS}}
{{.}} \
{{- end}}
2> "{{.TMP_ERR_LOG}}" \
| md5sum > "{{.CHECKSUM_FILE}}" \
; then
rm -f "{{.CHECKSUM_FILE}}"
{{- if eq "true" .FAIL}}
printf "[{{.TASK}} error] failed with:\n%s\n" "$(cat {{.TMP_ERR_LOG}})"
exit 1
{{- else}}
exit 0
{{- end}}
fi- This keeps your current behaviour (including logging and FAIL handling) but guarantees tar errors actually trigger the cleanup.
Optional hardening (if you want atomic writes and to avoid any transient bogus file): write the tar to a temp file, then hash that file into a temp checksum, then mv it into place. Happy to provide that variant, but it slightly changes how md5sum formats output (stdin prints “ -”, file prints the filename), so we’d also normalize the line to maintain compatibility.
- On ref: agreed—the loop-time “default (list) …” is redundant here because EXCLUDE_PATTERNS is already defaulted in the task vars. Also, for completeness: Task’s docs show ref expressions can call functions (examples use index and fromJson), so your current ref with default/list is valid; we just don’t need to repeat a default when the var is already normalized.
If you’re good with the minimal pipefail tweak, I’ll proceed with that.
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (1)
taskfiles/tests.yaml(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: unit-tests (macos-15)
- GitHub Check: unit-tests (ubuntu-24.04)
- GitHub Check: unit-tests (ubuntu-22.04)
🔇 Additional comments (1)
taskfiles/tests.yaml (1)
5-5: Include verified — default task presentConfirmed taskfiles/checksum/tests.yaml defines a
defaulttask (line 9).
| internal: true | ||
| cmds: | ||
| - task: "boost" | ||
| - task: "checksum" |
There was a problem hiding this comment.
🧩 Analysis chain
Broken task reference: calling an included Task requires a namespace or a wrapper.
- task: "checksum" will fail unless a local wrapper task named checksum exists. Match existing patterns (see remote, ystdlib-py) by adding a wrapper, or call the namespaced task directly.
Option A — add a wrapper (recommended for consistency with other entries):
tasks:
all:
internal: true
cmds:
- task: "boost"
- - task: "checksum"
+ - task: "checksum"
- task: "remote"
- task: "ystdlib-py"
+ checksum:
+ cmds:
+ - task: "checksum:default"Option B — call the namespaced default directly:
- - task: "checksum"
+ - task: "checksum:default"🏁 Script executed:
#!/bin/bash
# Validate that the "checksum" task is resolvable after your change.
# 1) List tasks to ensure the wrapper shows up (if using Option A).
task --list || true
# 2) Dry-run the "all" task to confirm it can reach checksum without executing commands.
task -n allLength of output: 85
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "PWD: $(pwd)"
echo
# helper to run rg if available, else grep fallback
if command -v rg >/dev/null 2>&1; then
echo "Using rg for searches"
RG_AVAIL=true
else
echo "rg not found; using grep fallback"
RG_AVAIL=false
fi
echo
echo "==> Check for taskfiles/tests.yaml"
if [ -f taskfiles/tests.yaml ]; then
echo "FOUND: taskfiles/tests.yaml (showing first 200 lines):"
nl -ba taskfiles/tests.yaml | sed -n '1,200p'
else
echo "taskfiles/tests.yaml not found at path taskfiles/tests.yaml"
fi
echo
echo "==> Search for occurrences of the word 'checksum' across the repo (context included)"
if [ "$RG_AVAIL" = true ]; then
rg -n --hidden -S '\bchecksum\b' -g '!node_modules/**' -g '!**/.git/**' -C3 || true
else
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E '\bchecksum\b' . || true
fi
echo
echo "==> Search for '- task:' entries (calls to included tasks)"
if [ "$RG_AVAIL" = true ]; then
rg -n --hidden -S '^\s*-\s*task\s*:' -g '!node_modules/**' -C3 || true
else
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E '^\s*-\s*task\s*:' . || true
fi
echo
echo "==> Search for 'checksum' task wrapper definitions (e.g. 'checksum:' or 'checksum:default')"
if [ "$RG_AVAIL" = true ]; then
rg -n --hidden -S '^\s*checksum\s*:' -g '!node_modules/**' -C3 || true
rg -n --hidden -S 'checksum:default' -g '!node_modules/**' -C3 || true
else
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E '^\s*checksum\s*:' . || true
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E 'checksum:default' . || true
fiLength of output: 179
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "PWD: $(pwd)"
echo
echo "==> List taskfiles directory"
ls -la taskfiles || true
echo
echo "==> Show taskfiles/tests.yaml (up to 400 lines) if present"
if [ -f taskfiles/tests.yaml ]; then
echo "----- taskfiles/tests.yaml start -----"
sed -n '1,400p' taskfiles/tests.yaml || true
echo "----- taskfiles/tests.yaml end -----"
else
echo "taskfiles/tests.yaml not found at taskfiles/tests.yaml"
fi
echo
echo "==> Search repository for occurrences of 'checksum' (3 lines context)"
if command -v rg >/dev/null 2>&1; then
rg -n --hidden -S '\bchecksum\b' -g '!node_modules/**' -g '!**/.git/**' -C3 || true
else
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E '\bchecksum\b' . || true
fi
echo
echo "==> Search for '- task:' entries (calls to included tasks) with context"
if command -v rg >/dev/null 2>&1; then
rg -n --hidden -S '^\s*-\s*task\s*:' -g '!node_modules/**' -C3 || true
else
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E '^\s*-\s*task\s*:' . || true
fi
echo
echo "==> Search for task wrapper/definition keys named 'checksum' or 'checksum:default'"
if command -v rg >/dev/null 2>&1; then
rg -n --hidden -S '^\s*checksum\s*:' -g '!node_modules/**' -C3 || true
rg -n --hidden -S 'checksum:default' -g '!node_modules/**' -C3 || true
else
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E '^\s*checksum\s*:' . || true
grep -RIn --line-number --exclude-dir=node_modules --exclude-dir=.git -I -n -E 'checksum:default' . || true
fiLength of output: 1913
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "PWD: $(pwd)"
echo
echo "==> Show taskfiles/checksum/tests.yaml (up to 400 lines)"
if [ -f taskfiles/checksum/tests.yaml ]; then
sed -n '1,400p' taskfiles/checksum/tests.yaml || true
else
echo "MISSING: taskfiles/checksum/tests.yaml"
fi
echo
echo "==> Show top-level keys in taskfiles/checksum/tests.yaml (yaml keys)"
if command -v python3 >/dev/null 2>&1; then
python3 - <<'PY'
import sys, yaml, pathlib
p = pathlib.Path("taskfiles/checksum/tests.yaml")
if not p.exists():
print("file not found")
sys.exit(0)
try:
data = yaml.safe_load(p.read_text())
except Exception as e:
print("yaml parse error:", e)
sys.exit(0)
if not isinstance(data, dict):
print("yaml root is not a mapping")
sys.exit(0)
for k in data.keys():
print(k)
# If there is a 'tasks' key, print its top-level task names
if 'tasks' in data and isinstance(data['tasks'], dict):
print("\nTop-level tasks in this file:")
for t in data['tasks'].keys():
print("-", t)
PY
else
echo "python3 not available; skipping YAML introspection"
fiLength of output: 4826
Fix broken task reference "checksum"
-
- task: "checksum" in taskfiles/tests.yaml does not resolve to a local task; the included namespace taskfiles/checksum/tests.yaml exposes a default task. Replace the call with a namespaced reference or add a local wrapper.
Option A — add a wrapper (consistent with other entries):
tasks:
all:
internal: true
cmds:
- task: "boost"
- - task: "checksum"
+ - task: "checksum"
- task: "remote"
- task: "ystdlib-py"
+ checksum:
+ cmds:
+ - task: "checksum:default"Option B — call the namespaced default directly:
- - task: "checksum"
+ - task: "checksum:default"Verified: taskfiles/checksum/tests.yaml defines a top-level "default" task.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - task: "checksum" | |
| tasks: | |
| all: | |
| internal: true | |
| cmds: | |
| - task: "boost" | |
| - task: "checksum" | |
| - task: "remote" | |
| - task: "ystdlib-py" | |
| checksum: | |
| cmds: | |
| - task: "checksum:default" |
🤖 Prompt for AI Agents
In taskfiles/tests.yaml around line 14, the task entry "- task: "checksum""
incorrectly references a non-existent local task; the included file
taskfiles/checksum/tests.yaml exposes a top-level "default" task. Fix by either
(A) adding a local wrapper task named "checksum" that delegates to the included
namespace’s default (consistent with other entries), or (B) replace the call
with the namespaced reference to the included file’s default task (e.g., use the
fully qualified namespace/task name). Ensure the chosen change matches the
repository convention for other entries and that task invocation resolves to the
checksum default task.
There was a problem hiding this comment.
I agree that we should add a wrapper task to stay consistent with remote:default and ystdlib-py:default.
There was a problem hiding this comment.
@Bill-hbrhbr, thank you for confirming! Adding the wrapper task will indeed maintain consistency with the existing patterns used for remote and ystdlib-py. The suggested code change above shows exactly how to implement this - it adds the local checksum task that delegates to checksum:default, matching the structure of the other test wrappers in the file.
| - |- | ||
| if ! cmp -s '{{.CHECKSUM_FILE}}' '{{.CHECKSUM_FILE_REF}}'; then | ||
| exit 0 | ||
| fi | ||
| exit 1 |
There was a problem hiding this comment.
| - |- | |
| if ! cmp -s '{{.CHECKSUM_FILE}}' '{{.CHECKSUM_FILE_REF}}'; then | |
| exit 0 | |
| fi | |
| exit 1 | |
| - "( cmp -s '{{.CHECKSUM_FILE}}' '{{.CHECKSUM_FILE_REF}}' && exit 2 ) || [ $? -eq 1 ]" |
The idea is that to only accept 1 as success. 0 when files are identical and >1 when file comparison has operational errors all results in failure.
| - |- | ||
| if ! cmp -s '{{.CHECKSUM_FILE}}' '{{.CHECKSUM_FILE_REF0}}'; then | ||
| exit 0 | ||
| fi | ||
| exit 1 |
There was a problem hiding this comment.
| - |- | |
| if ! cmp -s '{{.CHECKSUM_FILE}}' '{{.CHECKSUM_FILE_REF0}}'; then | |
| exit 0 | |
| fi | |
| exit 1 | |
| - "( cmp -s '{{.CHECKSUM_FILE}}' '{{.CHECKSUM_FILE_REF0}}' && exit 2 ) || [ $? -eq 1 ]" |
| - defer: "rm -f '{{.TMP_ERR_LOG}}'" | ||
| - |- | ||
| if ! \ | ||
| {{.TAR}} \ | ||
| --create \ | ||
| --file - \ | ||
| --group 0 \ | ||
| --mtime "UTC 1970-01-01" \ | ||
| --numeric-owner \ | ||
| --owner 0 \ | ||
| --sort name \ | ||
| --no-anchored \ | ||
| --wildcards \ | ||
| {{- range .EXCLUDE_PATTERNS}} | ||
| --exclude="{{.}}" \ | ||
| {{- end}} | ||
| {{- range .INCLUDE_PATTERNS}} | ||
| {{.}} \ | ||
| {{- end}} | ||
| 2> "{{.TMP_ERR_LOG}}" \ | ||
| | md5sum > "{{.CHECKSUM_FILE}}" \ | ||
| ; then |
There was a problem hiding this comment.
isn't the checksum file deleted if there's any failure within the tar->pipe->md5sum process? I don't think we need to worry about truncated checksums.
Also ref: "default (list) .EXCLUDE_PATTERNS" is redundant. We already do this in the vars: section.
| - defer: "rm -f '{{.TMP_CHECKSUM_FILE}}'" | ||
| # Check that all paths exist and the checksum matches; otherwise delete the checksum file. | ||
| ref: ".EXCLUDE_PATTERNS" | ||
| FAIL: "false" |
There was a problem hiding this comment.
| FAIL: "false" | |
| ignore_error: true |
Can we use ignore_error = true here and avoid introducing a new variable for checksum:compute? I don’t know the other cases where we’ll need to allow failures.
There was a problem hiding this comment.
Sorry, the motivation is confusing since I split the PRs. This is necessary for better checksum workflows like in boost. I'll improve the description.
https://github.com/y-scope/yscope-dev-utils/pull/80/files#diff-a1f9d497b81cb7bdb53e553d8fce46fbee451bbe4629106955d2536ada79015dR116
In this case when we validate the source directory and it fails we want to completely fail.
One option is to just assume this cannot fail and not check it, but that seems a bit much to avoid a flag in rare cases.
The other option is to try and recover, but that is very complicated. We'd need boost:build-and-install to somehow re-run boost:generate, which is a bit ugly/complicated. More generally any tasks that implicitly depends on another task (but cannot be deps without a lot of work due to their library/helper style) will need to support a recovery pass in certain cases.
| TAR: "{{if eq OS \"darwin\"}}gtar{{else}}tar{{end}}" | ||
| TMP_ERR_LOG: "{{.CHECKSUM_FILE}}.log.tmp" |
There was a problem hiding this comment.
we will improve this once we add Linux/Alphine CIs
| internal: true | ||
| cmds: | ||
| - task: "boost" | ||
| - task: "checksum" |
There was a problem hiding this comment.
I agree that we should add a wrapper task to stay consistent with remote:default and ystdlib-py:default.
| taskfile: "../../exports/taskfiles/utils/checksum.yaml" | ||
|
|
||
| tasks: | ||
| default: |
There was a problem hiding this comment.
should we add tests for testing FAIL flags for checksum:validate?
There was a problem hiding this comment.
I agree with you and want to, but afaik we cannot actually test if a task fails using task itself. E.g. if FAIL is true and the task fails, the entire task "run" will stop and exit.
Essentially, (afaik) there is no way to write a unit test (in task) that verifies a task fails.
We probably need to switch to using python to test tasks eventually.
There was a problem hiding this comment.
right. for now we can only manually test.
Co-authored-by: Bingran Hu <[email protected]>
2 participants
Description
This PR addresses a few issues with the
checksumtasks and also adds unit tests for these tasks.checksum:computerequires gnu tar for its flags and would fail silently on MacOS (unlesstarwas set to gnu tar)gtarif running on macos.FAILflag allowing users to change whether they wish to allow the task to fail or not.Checklist
breaking change.
Validation performed
Summary by CodeRabbit
New Features
Documentation
Tests
Chores