Skip to content

fix: redirect Crush npm install to writable RUNNER_TEMP prefix to fix EROFS failures#29468

Closed
Copilot wants to merge 2 commits intomainfrom
copilot/aw-failures-fix-smoke-engines
Closed

fix: redirect Crush npm install to writable RUNNER_TEMP prefix to fix EROFS failures#29468
Copilot wants to merge 2 commits intomainfrom
copilot/aw-failures-fix-smoke-engines

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 1, 2026

@charmland/crush skips its native binary download when installed with --ignore-scripts, then lazily creates bin/ inside the package directory on first invocation. With npm install -g, the package lands in /opt/hostedtoolcache/node/.../lib/node_modules/ — read-only on newer GitHub Actions runners — causing EROFS: read-only file system failures in the agent job's "Execute Crush CLI" step.

Changes

crush_engine.go

  • Added crushGlobalPrefix = "${RUNNER_TEMP}/gh-aw/crush-global" constant
  • Replaced BuildStandardNpmEngineInstallSteps with a new buildCrushInstallStep that installs with an explicit writable --prefix and appends the bin dir to $GITHUB_PATH:
    CRUSH_PREFIX="${RUNNER_TEMP}/gh-aw/crush-global"
    mkdir -p "${CRUSH_PREFIX}"
    npm install --ignore-scripts --global --prefix "${CRUSH_PREFIX}" @charmland/crush@0.59.0
    echo "${CRUSH_PREFIX}/bin" >> "${GITHUB_PATH}"
  • In the AWF (firewall) execution path, added an explicit export PATH="${RUNNER_TEMP}/gh-aw/crush-global/bin:$PATH" inside the container command — necessary because sudo's secure_path strips $GITHUB_PATH additions (same pattern as GetMCPCLIPathSetup)

crush_engine_test.go

  • Added findInstallStep helper to locate the step by name rather than fragile index lookup
  • Assertions for writable prefix, --prefix flag, $GITHUB_PATH addition, and crush bin PATH in AWF execution step

Lock files

  • All 205 workflow lock files recompiled

Out of scope

The Smoke Gemini API_KEY_INVALID failure is purely operational — the GEMINI_API_KEY repository secret has expired and needs to be rotated. The engine code is correct.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw Kazdp_E-aCT60Thr-C (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw rt/yaml (http block)
  • https://api.github.com/orgs/test-owner/actions/secrets
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name -json GOMOD 64/bin/go -d GO111MODULE 64/bin/go Bb3BnF9Rs13V env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /orgs/test-owner/actions/secrets --jq .secrets[].name 38965057/001' 38965057/001' eport.lock.yml --show-toplevel git /usr/bin/git git ortc�� 7365980c12808757474442ea:pkg/workflow/crush_engine_test.go stmain.go ache/go/1.25.8/x64/pkg/tool/linux_amd64/link --show-toplevel git /usr/bin/git ache/go/1.25.8/x64/pkg/tool/linuconfig (http block)
  • https://api.github.com/repos/actions/ai-inference/git/ref/tags/v1
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq [.object.sha, .object.type] | @tsv GOMODCACHE 64/pkg/tool/linux_amd64/vet /usr/bin/git -json GO111MODULE ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel go /usr/bin/git /ref/tags/v9 GO111MODULE x_amd64/compile git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/ai-inference/git/ref/tags/v1 --jq [.object.sha, .object.type] | @tsv xterm-color git /usr/bin/git y_with_repos=pubgit rev-parse me: String!) { --show-toplevel git rev-�� --show-toplevel git /usr/bin/git agent-performancnode config kflows/schema-feinstall git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v3
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv k/gh-aw/gh-aw/.github/workflows/artifacts-summary.md go /usr/bin/git -json GO111MODULE 64/bin/go git rev-�� --show-toplevel go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v3 --jq [.object.sha, .object.type] | @tsv /home/REDACTED/work/gh-aw/gh-aw/.github/workflows (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v5
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv 3958-37337/test-3515752513/.github/workflows stmain.go ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet env -json GO111MODULE /opt/hostedtoolcache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel ortcfg /usr/bin/git g/envutil/envutigit g/envutil/envutirev-parse ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/compile /usr/bin/git 8297654/b417/_pkgit GO111MODULE 8297654/b417=> git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v5 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/git --show-toplevel /usr/bin/php8.3 /usr/bin/git git rev-�� --show-toplevel git /usr/bin/git --show-toplevel go /usr/bin/git git (http block)
  • https://api.github.com/repos/actions/checkout/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ignore-path ../../../.prettierignore (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv */*.ts' '**/*.jsremote.origin.url 773105/b003/vet.cfg ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv */*.ts' '**/*.js-f 773105/b072/vet.owner=github cfg (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v8
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv INVALID,NEW -j DROP -json GO111MODULE eutil.test git rev-�� --show-toplevel eutil.test /usr/bin/git -json GO111MODULE /opt/hostedtoolc-t git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v8 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet /usr/bin/git /home/REDACTED/worgit config ache/go/1.25.8/x64/pkg/tool/linu--show-toplevel git rev-�� --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/vet /usr/bin/git /home/REDACTED/worgit rev-parse 066423/b362/vet.--show-toplevel git (http block)
  • https://api.github.com/repos/actions/github-script/git/ref/tags/v9
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOSUMDB GOWORK 64/bin/go sh -c "prettier" --cheGOINSECURE bash 64/bin/go --noprofile ache/go/1.25.8/xenv 64/bin/go sed (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv npx prettier --cGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go sh -c "prettier" --cheGOINSECURE bash 64/bin/go tierignore ache/go/1.25.8/x--norc 64/bin/go go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/github-script/git/ref/tags/v9 --jq [.object.sha, .object.type] | @tsv prettier --check 64/bin/go **/*.ts **/*.json --ignore-path go list�� -e -f 64/bin/go -- unsafe 64/bin/go go (http block)
  • https://api.github.com/repos/actions/setup-go/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv inputs.version go /usr/bin/infocmp -json GO111MODULE ndor/bin/bash infocmp -1 xterm-color go /usr/bin/git 52228650/001 GO111MODULE util.test git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-go/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel git /usr/bin/infocmp ithub/workflows erena-mcp-serverrev-parse /usr/bin/infocmp--show-toplevel infocmp -1 xterm-color infocmp /usr/bin/git 2222522665/.githgit node /usr/bin/git git (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv ry=1 s/4/artifacts 8297654/b462/_pkg_.a -json GO111MODULE 64/bin/go gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq ache/node/24.14.1/x64/bin/node -json GO111MODULE 64/bin/go ache/node/24.14.1/x64/bin/node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv --show-toplevel go /usr/bin/git LsRemoteWithRealgit LsRemoteWithRealrev-parse ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel go /usr/bin/git 52228650/001 GO111MODULE ache/go/1.25.8/x--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv /tmp/TestHashStability_SameInputSameOutput483906994/001/stability-test.md s/2/artifacts (http block)
  • https://api.github.com/repos/actions/setup-node/git/ref/tags/v6
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv ignore-path ../../../.prettierignore (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv */*.ts' '**/*.json' --ignore-patremote.origin.url 773105/b071/vet.cfg 1/x64/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin/node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/setup-node/git/ref/tags/v6 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE /snap/bin/bash GOINSECURE GOMOD GOMODCACHE bash --no�� licyBlockedUsersCommaSeparatedCompiledOutput1522322766/001 GOPROXY /usr/bin/git GOSUMDB GOWORK 64/bin/go git (http block)
  • https://api.github.com/repos/actions/upload-artifact/git/ref/tags/v4
    • Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv /tmp/TestGuardPolicyMinIntegrityOnlymin-integrity_with_repos_array_c1063529660/001 remote ache/node/24.14.1/x64/bin/node /etc/php/8.3/cgigit -d 64/bin/go infocmp t-20�� k/gh-aw/gh-aw/.github/workflows/ai-moderator.md go /usr/bin/git -json GO111MODULE 64/bin/go git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/actions/upload-artifact/git/ref/tags/v4 --jq [.object.sha, .object.type] | @tsv ons-test2761433864 --json ache/node/24.14.1/x64/bin/node --repo owner/repo r: $owner, name:--show-toplevel ache/node/24.14.1/x64/bin/node -267�� /usr/bin/git git /usr/bin/git 7365980c12808757git git yml git (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv h ../../../.prettierignore (http block)
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv h ../../../.pret.prettierignore (http block)
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv --show-toplevel gh /usr/bin/git download 5 /usr/bin/git git rev-�� --show-toplevel git /usr/lib/git-cor168.63.129.16 -L current (local capi /usr/bin/git /usr/lib/git-cor-f (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch twS7/KogfOlBF5askdAIFtwS7 (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch --show-toplevel git 64/pkg/tool/linux_amd64/asm .1/x64/codeql/toinfocmp /opt/hostedtoolc-1 /usr/bin/git 64/pkg/tool/linux_amd64/asm rev-�� --show-toplevel git repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } --show-toplevel /opt/hostedtoolc-C /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v0.1.2
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq [.object.sha, .object.type] | @tsv --show-toplevel go /usr/bin/git mLsRemoteWithReagit mLsRemoteWithRearev-parse 64/bin/go /usr/bin/git remo�� -v go /usr/bin/git mpiledOutput2031git GO111MODULE ache/go/1.25.8/x--show-toplevel git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v0.1.2 --jq [.object.sha, .object.type] | @tsv --show-toplevel docker /usr/bin/infocmp hub/workflows git kflows/daily-tea--show-toplevel infocmp -1 xterm-color git /usr/bin/git ithub/workflows config 1/x64/bin/bash git (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv /tmp/TestGuardPolicyBlockedUsersCommaSeparatedCompiledOutput1522322766/001 rev-parse ache/node/24.14.1/x64/bin/node -ignore_readdir_git -lname 64/bin/go git t-32�� k/gh-aw/gh-aw/.github/workflows/ai-moderator.md remote /usr/bin/git -json GO111MODULE 64/bin/go git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv /tmp/gh-aw-test-runs/20260501-084621-47425/test-1478354603/.github/workflows rev-parse cfg --show-toplevel git r: $owner, name:--show-toplevel git remo�� /usr/bin/git l /opt/hostedtoolcache/node/24.14.1/x64/bin/node ithub/workflows git 4312494/b474/vet--show-toplevel node (http block)
  • https://api.github.com/repos/github/gh-aw-actions/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv /tmp/TestGuardPolicyBlockedUsersExpressionCompiledOutput3642587234/001 remote /usr/bin/git -ignore_readdir_git -lname 64/bin/go git -C /tmp/TestCompileUpdateDiscussionFieldEnforcement1740723468/001 rev-parse r,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,disp--show-toplevel -json GO111MODULE 64/bin/go node (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw-actions/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv /tmp/TestHashStability_SameInputSameOutput483906994/001/stability-test.md gqXZfEv3XNby /usr/bin/git ithub/workflows git repository(owne--show-toplevel git rev-�� --show-toplevel git me: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } --show-toplevel git rostylelite-mark--show-toplevel gh (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-24 GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-04-01 GOMOD GOMODCACHE go env add-source-path-1079346944 GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --limit 100 --created >=2026-01-31 GOMOD GOMODCACHE go env ned-imports-enabled-with-env-template-expressions-in-body.md GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/1/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/1/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linu-nolocalimports GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linu/tmp/go-build1818297654/b462/_testmain.go env util.test GO111MODULE ortcfg.link GOINSECURE GOMOD GOMODCACHE 6lOae4WsPt2nvzZ6yM/kanFXMcb9Ib9j-importcfg (http block)
    • Triggering command: /usr/bin/gh gh run download 1 --dir test-logs/run-1 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/1/artifacts --jq .artifacts[].name :latest 1/x64/bin/bash --show-toplevel (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12345/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12345/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 12345 --dir test-logs/run-12345 GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linutest@example.com env 4114552232/.github/workflows GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12345/artifacts --jq .artifacts[].name git me: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } --show-toplevel git /usr/bin/git infocmp -1 01/main.md erena-mcp-server:latest (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/12346/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12346/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/compile env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 12346 --dir test-logs/run-12346 GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE Vgol9MA/XeIrSce2aWTPZmVH_ghz env 4114552232 GO111MODULE x_amd64/vet GOINSECURE ole GOMODCACHE x_amd64/vet (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/12346/artifacts --jq .artifacts[].name rev-parse /usr/bin/gh l git /usr/bin/git /usr/bin/gh api ithub/workflows -f /usr/bin/infocmp -f owner=github DiscussionsEnabl--show-toplevel infocmp (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/2/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/2/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linutest@example.com env 159908349/.github/workflows GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 2 --dir test-logs/run-2 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env 100902451/custom/workflows GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/2/artifacts --jq .artifacts[].name show /usr/bin/git --show-toplevel /opt/hostedtoolc-1 ptables git -C ut1174186434/001 show 64/pkg/tool/linux_amd64/vet ne.go git /usr/bin/git 64/pkg/tool/linux_amd64/vet (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/3/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/3/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet env -json GO111MODULE k GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run download 3 --dir test-logs/run-3 GO111MODULE 64/pkg/tool/linux_amd64/compile GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linutest@example.com env 100902451 GO111MODULE ck GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/3/artifacts --jq .artifacts[].name :latest 86_64/bash remote.origin.urgh 'origin' iptables git -C ut1174186434/001--limit rver /usr/bin/git remote.origin.urgit git $name) { has--show-toplevel git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/4/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/4/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/vet GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linuTest User env 159908349/.github/workflows GO111MODULE x_amd64/compile GOINSECURE GOMOD GOMODCACHE x_amd64/compile (http block)
    • Triggering command: /usr/bin/gh gh run download 4 --dir test-logs/run-4 GO111MODULE 64/pkg/tool/linu-importcfg GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linu/home/REDACTED/work/gh-aw/gh-aw/pkg/typeutil/convert_test.go env util.test GO111MODULE ortcfg.link GOINSECURE GOMOD GOMODCACHE iUEqf5PFeb3NCkL0nF/2rw-RdHCw_apHrev-parse (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/4/artifacts --jq .artifacts[].name mLsRemoteWithRealGitbranch_with_hyphen1736223284/001' /usr/bin/gh l /opt/hostedtoolcremote es /usr/bin/gh api ut1174186434/001 -f /usr/bin/git -f owner=github -f git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/runs/5/artifacts
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/5/artifacts --jq .artifacts[].name GO111MODULE 64/pkg/tool/linux_amd64/link GOINSECURE GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/link env 159908349/.github/workflows GO111MODULE ortcfg.link GOINSECURE GOMOD GOMODCACHE UB4Z85HNraJd7B2VBq/aMwv_nfK75JUCd43tsyO/DAaZteIL-3lxG0HoYBiF (http block)
    • Triggering command: /usr/bin/gh gh run download 5 --dir test-logs/run-5 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env plorer.md GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api --paginate repos/{owner}/{repo}/actions/runs/5/artifacts --jq .artifacts[].name config 64/bin/bash remote.origin.ur/usr/bin/git (http block)
  • https://api.github.com/repos/github/gh-aw/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path -json GOCACHE 64/bin/go -d GO111MODULE 64/bin/go go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 100 GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run list --json databaseId,number,url,status,conclusion,workflowName,createdAt,startedAt,updatedAt,event,headBranch,headSha,displayTitle --workflow nonexistent-workflow-12345 --limit 6 GOMOD GOMODCACHE 64/pkg/tool/linux_amd64/vet env 1114831460 GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/contents/.github/workflows/shared/reporting.md
    • Triggering command: /tmp/go-build1818297654/b404/cli.test /tmp/go-build1818297654/b404/cli.test -test.testlogfile=/tmp/go-build1818297654/b404/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true GOINSECURE GOMOD GOMODCACHE 8996881/b434/impGO111MODULE /hom�� che/go-build/db/GOINSECURE **/*.cjs 64/bin/go **/*.json --ignore-path ../../../.pretti-json /opt/hostedtoolcGO111MODULE (http block)
    • Triggering command: /tmp/go-build511066423/b404/cli.test /tmp/go-build511066423/b404/cli.test -test.testlogfile=/tmp/go-build511066423/b404/testlog.txt -test.paniconexit0 -test.v=true -test.parallel=4 -test.timeout=10m0s -test.run=^Test -test.short=true --show-toplevel git /usr/bin/git ache/go/1.25.8/x-f rev-�� --show-toplevel git r: $owner, name:-f --show-toplevel /usr/lib/git-cor-C /usr/bin/git git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v0.47.4
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq [.object.sha, .object.type] | @tsv --show-toplevel ache/go/1.25.8/xtest@example.com /usr/bin/git 3958-37337/test-git GO111MODULE ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel ache/go/1.25.8/x64/pkg/tool/linuremote.origin.url /usr/bin/git 8297654/b437/_pkgit GO111MODULE 8297654/b437=> git (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v0.47.4 --jq [.object.sha, .object.type] | @tsv --show-toplevel /usr/bin/gh /usr/bin/git 4621-47425/test-git tmain.go ache/go/1.25.8/x--show-toplevel git rev-�� --show-toplevel ache/go/1.25.8/x64/pkg/tool/linux_amd64/link /usr/bin/git 066423/b404/cli.du rev-parse e/git git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE /opt/hostedtoolcache/go/1.25.8/x-test.short=true GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.0.0 --jq [.object.sha, .object.type] | @tsv 3952732187 ghcr.io/github/serena-mcp-server:latest 64/pkg/tool/linux_amd64/vet --show-toplevel infocmp /usr/bin/git 64/pkg/tool/linux_amd64/vet /opt�� view -- /usr/bin/docker name /usr/bin/gh /usr/bin/git docker (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v1.2.3
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v1.2.3 --jq [.object.sha, .object.type] | @tsv ithub/workflows git /opt/hostedtoolcache/go/1.25.8/x64/pkg/tool/linux_amd64/vet --show-toplevel /opt/hostedtoolc/tmp/js-hash-test-833145996/test-hash.js /usr/bin/git /opt/hostedtoolcache/go/1.25.8/xconfig -ato�� k/gh-aw/gh-aw/.github/workflows -buildtags rt.lock.yml l -ifaceassert -nilfunc git (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v2.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go 9853�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v2.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env lGitmaster_branch3985388388/001' lGitmaster_branch3985388388/001' 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
  • https://api.github.com/repos/github/gh-aw/git/ref/tags/v3.0.0
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq [.object.sha, .object.type] | @tsv -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go 4701�� -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw/git/ref/tags/v3.0.0 --jq [.object.sha, .object.type] | @tsv --show-toplevel gh yml /repos/actions/cgit --jq /usr/bin/git /opt/hostedtoolcache/go/1.25.8/xshow -ato�� -bool -buildtags ock.yml -errorsas -ifaceassert -nilfunc git (http block)
  • https://api.github.com/repos/nonexistent/action/git/ref/tags/v999.999.999
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq [.object.sha, .object.type] | @tsv or.md GO111MODULE ck GOINSECURE GOMOD GOMODCACHE go sRem�� -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/nonexistent/action/git/ref/tags/v999.999.999 --jq [.object.sha, .object.type] | @tsv xterm-color /lib/jspawnhelpe-test.timeout=10m0s /home/REDACTED/.local/bin/node --show-toplevel infocmp /usr/bin/git node /opt�� k/gh-aw/gh-aw/.github/workflows -- cfg l Test User /usr/bin/git git (http block)
  • https://api.github.com/repos/nonexistent/repo/actions/runs/12345
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion GOINSECURE GOMOD GOMODCACHE eutil.test 8182�� -json GO111MODULE Name,createdAt,startedAt,updated-lang=go1.25 GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh run view 12345 --repo nonexistent/repo --json status,conclusion -f owner=github ed } } git -C k/gh-aw/gh-aw/.github/workflows config /usr/bin/docker remote.origin.urgit gh /usr/bin/git docker (http block)
  • https://api.github.com/repos/owner/repo/actions/workflows
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo -importcfg /tmp/go-build1818297654/b434/importcfg -pack /home/REDACTED/work/gh-aw/gh-aw/pkg/repoutil/repoutil.go /home/REDACTED/work/gh-aw/gh-aw/pkg/repoutil/repoutil_test.go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh workflow list --json name,state,path --repo owner/repo -nolocalimports -importcfg /tmp/go-build1818297654/b435/importcfg -pack /tmp/go-build1818297654/b435/_testmain.go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh workflow list --repo owner/repo --json name,path,state ache/go/1.25.8/x64/bin/go GOINSECURE tants GOMODCACHE go env 668/001/stability-test.md GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE ortcfg (http block)
  • https://api.github.com/repos/test-owner/test-repo/actions/secrets
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name GOPATH go 64/bin/go -d GO111MODULE 64/bin/go go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/test-owner/test-repo/actions/secrets --jq .secrets[].name --show-toplevel git me: String!) { repository(owner: $owner, name:-f --show-toplevel git /usr/bin/git git rev-�� ithub/workflows git r: $owner, name: $name) { hasDiscussionsEnabled } } --show-toplevel git /usr/bin/git ache/go/1.25.8/xshow (http block)
  • https://api.github.com/repos/test/repo
    • Triggering command: /usr/bin/gh gh api /repos/test/repo --jq .default_branch 19/001/test-simple-frontmatter.m-s GO111MODULE kflow.lock.yml GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE ache/go/1.25.8/x64/bin/go GOINSECURE GOMOD GOMODCACHE go (http block)
    • Triggering command: /usr/bin/gh gh api /repos/test/repo --jq .default_branch xterm-color gh /usr/bin/git /repos/actions/agit --jq /usr/bin/git git -C k/gh-aw/gh-aw/.github/workflows show /usr/bin/infocmp --show-toplevel git x_amd64/vet infocmp (http block)

If you need me to access, download, or install something from one of these locations, you can either:

… writable RUNNER_TEMP

The @charmland/crush npm package lazily installs its native binary to bin/ within
the package directory on first run. With `npm install -g`, the package ends up in
/opt/hostedtoolcache/node/.../lib/node_modules/ which is read-only on newer GitHub
Actions runners, causing EROFS failures.

Fix:
- Add `crushGlobalPrefix` constant pointing to ${RUNNER_TEMP}/gh-aw/crush-global
- Replace BuildStandardNpmEngineInstallSteps with a custom buildCrushInstallStep
  that uses `--prefix "${RUNNER_TEMP}/gh-aw/crush-global"` to redirect the npm
  global installation to a writable directory, and appends the bin dir to $GITHUB_PATH
- In the AWF (firewall) execution path, add explicit PATH setup for the crush bin
  dir so it's found inside the AWF container (sudo secure_path strips $GITHUB_PATH)
- Update crush_engine_test.go with assertions for writable prefix and PATH setup
- Recompile all 205 workflow lock files

The Gemini API_KEY_INVALID failure is a purely operational issue (expired secret)
with no code-level fix possible.

Closes #29458

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/ce69b917-3a39-4558-8762-08e75bb2785a

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix smoke engine failures for Gemini and Crush fix: redirect Crush npm install to writable RUNNER_TEMP prefix to fix EROFS failures May 1, 2026
Copilot AI requested a review from gh-aw-bot May 1, 2026 08:48
@pelikhan pelikhan added the awf label May 1, 2026
@pelikhan pelikhan closed this May 2, 2026
@github-actions github-actions Bot deleted the copilot/aw-failures-fix-smoke-engines branch May 9, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw-failures] [P0] Smoke Engine Failures: Gemini API_KEY_INVALID + Crush EROFS Install

3 participants