Onboard build failure analysis workflow

[YuliiaKovalova]

Summary

• Onboard Arcade's build-failure analysis agent and automatic/manual agentic workflows for Runtime's Azure Pipelines definition (129).
• Select failed/canceled-job Logs_Build_* artifacts by Azure DevOps' producing-job ID, with bounded extraction and hlx task-log fallback.
• Use Runtime's existing PAT pool, digest-pinned binlog tooling, deterministic PR targets, and revision/latest-build checks before publishing.
• Preserve queued commands, match activation exactly, and deduplicate answered command-comment URLs using persisted analysis metadata.

Validation

Both workflows compile with pinned gh-aw v0.86.2 using --strict --validate --schedule-seed dotnet/runtime. All 13 source-driven regression tests pass.

Fresh fork proof uses source cba969a, harness 6fcb943, and mirror PR #2. Retained Runtime build 1600462 supplies the real artifacts and compiler failure. Fresh selection/staging produced 14 binlogs from 7 of 102 log artifacts.

Case	Run	Verified result
Automatic check-run replay	35230857676	Real binlog_errors response containing CS1061; published diagnosis and inline suggestion at the mirrored head.
Real command with metadata-omission injection	35234194343	Concrete binlog calls; published diagnosis. After the harness removed both .data and its rendered block, the production step restored exactly one block on the summary and two inline items, preserving content and targets.
Edit the same answered command URL	35238614074	Agent and safe-output jobs skipped; no new analysis or inline comments.

Earlier fault/queue coverage is retained only for unchanged paths: actual upload/download failures, partial-file cleanup, malformed commands, queued-command survival, stale revisions, and retry after a pre-analysis failure. The changed producing-job association and NUL-delimited traversal have current regressions and fresh staging evidence.

Scope and limitations

• Automatic triggering was event replay, not genuine GitHub App webhook delivery. Fork adapters handle mirror identity, checkout restrictions, and ephemeral Actions-token inference; production PAT-pool authentication is unchanged.
• This is execution/output-path proof, not a clean sweep of model-quality attempts. One current-source command attempt produced a probe (test body) and failed acceptance. Its single retry included explicit binary-analysis/no-probe guidance.
• Automatic analysis queried one of 14 supplied binlogs. The accepted command made 15 concrete-path requests covering all 14, but only six successful backend responses were recorded. Complete successful per-file analysis and reliable bare-command behavior are not established.
• The compiler diagnostic and suggestion locations were checked; generated code suggestions remain advisory and were not applied or compiled. No upstream Azure build was manually rerun.
• The fork's original default main and SHA were restored. No personal credential was uploaded or added as an Actions secret.

Other output examples

dotnet/arcade#17348 (comment)
microsoft/testfx#10637 (comment)

This description was updated with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
15 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR adds an agentic GitHub Actions workflow pair for analyzing Azure Pipelines (runtime, definitionId 129) build failures by downloading the failed/canceled legs’ Logs_Build_* artifacts, extracting *.binlog, and delegating analysis to a repository-scoped build-failure analyst agent that reads the binlogs via the binlog-mcp MCP server and posts results via safe-outputs.

Changes:

• Introduces a shared prompt body and an agent playbook for consistent build-failure analysis behavior.
• Adds two workflows: an automatic check_run-triggered workflow and a maintainer /analyze-build-failure slash-command workflow, both reusing failed-job binlogs instead of rebuilding.
• Pins the binlog-mcp container digest in the actions lock data and includes the compiled .lock.yml output.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
.github/workflows/shared/build-failure-analysis-shared.md	Shared agent prompt body imported by the two workflows.
.github/workflows/build-failure-analysis.md	Automatic workflow that fetches failed/canceled job binlogs from ADO and runs the analysis agent with safe-outputs.
.github/workflows/build-failure-analysis.lock.yml	gh-aw compiled lock workflow (generated) with pinned actions/containers and emitted jobs.
.github/workflows/build-failure-analysis-command.md	Slash-command workflow to rerun analysis on a PR’s latest failed build.
.github/aw/actions-lock.json	Adds a pinned container digest entry for the binlog-mcp image.
.github/agents/build-failure-analyst.agent.md	Adds the reusable build-failure analyst agent playbook used by the workflows.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/build-failure-analysis.md:130

• mcp-servers.binlog-mcp.allowed: ["*"] grants the agent access to every tool the binlog-mcp MCP server exposes. Even with a digest-pinned container, this is a broader capability surface than necessary and makes future binlog-mcp tool additions automatically callable by the agent. Consider narrowing this allowlist to only the specific binlog_* tools the analyst is expected to use (and/or a supported wildcard that only matches binlog_*).

  binlog-mcp:
    container: "mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-binlog-mcp-amd64"
    mounts:
      - "/tmp/binlogs:/data/binlogs:ro"
    allowed: ["*"]

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/workflows/build-failure-analysis.md:519

• The per-artifact binlog staging counter i is only incremented on a successful cp. If one cp fails and the artifact contains multiple *.binlog files, the next file will reuse the same destination name (e.g. ${ai}_0_...) and can overwrite a previously staged binlog, contradicting the "destinations unique" comment and potentially hiding the real failing leg.

# on `needs.fetch-binlog.outputs.binlog-found == 'true'`, these only run once
# binlogs have been retrieved from the failed Azure DevOps build.
steps:
  - name: Download analysis artifact
    uses: actions/download-artifact@v8.0.1

.github/workflows/build-failure-analysis-command.md:460

• The per-artifact binlog staging counter i is only incremented when cp succeeds. If an artifact contains multiple *.binlog files and an early copy fails, subsequent files can reuse the same ${ai}_${i}_... destination name and overwrite staged data, which can make the analysis incomplete or misleading.

            # Info-ZIP prepends warnings on STDOUT for a recoverable archive,
            # and a multi-line value would still pass the `grep -qE` check
            # below, since `grep -q` matches if ANY line matches. `timeout`
            # bounds a hostile archive; pipefail + fail-closed because a killed
            # probe's partial output can end in a numeric column and undercount.

[YuliiaKovalova]

Addressed both review comments in b785d9c by using sanitized artifact names for every workflow-command warning. A subsequent fork E2E run also exposed and fixed the generated safe-output target expression in 0e84cd0. Full E2E proof: https://github.com/YuliiaKovalova/runtime/actions/runs/32471218479

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.

[YuliiaKovalova]

🤖 This update was generated with GitHub Copilot.

Addressed the Runtime artifact-layout feedback in e3024a72fdb:

• Keeps binlog-mcp as the primary analyzer and adds digest-pinned lewing/helix.mcp v0.8.0 as a read-only Azure DevOps compile-log fallback. Only azdo_timeline, azdo_search_timeline, azdo_search_log, azdo_log, and azdo_artifacts are exposed; Helix/test diagnosis remains out of scope.
• Matches Logs_Build_* artifacts exact-first, then accepts a prefix only when it identifies one timeline job. This covers Runtime's display-only monointerpreter, minijit, and llvmaot suffixes without confusing similarly named jobs.
• Runs analysis even when a failed compile task has no matching binlog, inventories failed timeline tasks, and uses bounded hlx evidence for anything not explained by binlogs or complete timeline diagnostics.
• Makes per-artifact staging all-or-nothing, fixes destination reuse after copy failures, and revalidates both PR head and merge revisions before analysis and immediately before safe-output writes.

Fork E2E: https://github.com/YuliiaKovalova/runtime/actions/runs/33157134057

The run replayed Runtime build 1561838 and selected 15 of 68 build-log artifacts (previous exact matching selected 9), including all six display-only suffix cases. It staged 74 binlogs from all 15 artifacts, used hlx timeline evidence for the unmatched Build managed test components/compile failures, and posted exactly one summary plus one review suggestion pinned to commit d297327fa0b20bbabed8b4ee203885303dadab70:

• Summary: E2E target for build failure analysis YuliiaKovalova/runtime#1 (comment)
• Pinned suggestion: E2E target for build failure analysis YuliiaKovalova/runtime#1 (comment)

Both source workflows compile cleanly with gh aw compile --strict --validate on gh-aw v0.86.2. The two follow-up review comments about stale activation documentation are fixed and resolved in the same head.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis.md:487

• The PIPESTATUS checks after unzip -Z1 | grep -qE ... can misclassify the "suspicious path" case as an unzip failure. When grep -q matches early it closes the pipe and unzip may exit non-zero due to SIGPIPE; checking zscan_rc[0] first will then hit the "could not list" branch instead of the intended "suspicious entry path" branch.

            if [ "${zscan_rc[0]}" -ne 0 ]; then
              echo "::warning::Skipping ${safe_name}: could not list archive entries (unzip -Z1 rc=${zscan_rc[0]})."; continue
            fi
            if [ "${zscan_rc[1]}" -eq 0 ]; then
              echo "::warning::Skipping ${safe_name}: archive has a suspicious (absolute or ..) entry path."; continue

.github/workflows/build-failure-analysis-command.md:541

• The PIPESTATUS checks after unzip -Z1 | grep -qE ... can misclassify the "suspicious path" case as an unzip failure. When grep -q matches early it closes the pipe and unzip may exit non-zero due to SIGPIPE; checking zscan_rc[0] first will then hit the "could not list" branch instead of the intended "suspicious entry path" branch.

            if [ "${zscan_rc[0]}" -ne 0 ]; then
              echo "::warning::Skipping ${safe_name}: could not list archive entries (unzip -Z1 rc=${zscan_rc[0]})."; continue
            fi
            if [ "${zscan_rc[1]}" -eq 0 ]; then
              echo "::warning::Skipping ${safe_name}: archive has a suspicious (absolute or ..) entry path."; continue

[Copilot]

Pull request overview

Copilot reviewed 5 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis.md:482

• The comment says anything left in /tmp/binlogs from an earlier run would otherwise be uploaded and attributed to this build, but the cleanup only removes *.binlog. Any other leftover files would still be uploaded with the artifact. Consider clearing the directory contents instead of only binlogs to match the stated invariant.

          # Only binlogs extracted by this run may be analyzed. Anything left in
          # the directory by an earlier run on the same runner would otherwise be
          # uploaded and attributed to this build.
          rm -f /tmp/binlogs/*.binlog

.github/workflows/build-failure-analysis-command.md:537

• The comment says anything left in /tmp/binlogs from an earlier run would otherwise be uploaded and attributed to this build, but the cleanup only removes *.binlog. Any other leftover files would still be uploaded with the artifact. Consider clearing the directory contents instead of only binlogs to match the stated invariant.

          # Only binlogs extracted by this run may be analyzed. Anything left in
          # the directory by an earlier run on the same runner would otherwise be
          # uploaded and attributed to this build.
          rm -f /tmp/binlogs/*.binlog

[Copilot]

🔵 Needs a closer look

Multiple moderate workflow correctness, validation, fallback, and safe-output issues remain unresolved.

Review details

Suppressed comments (15)

Previously missed (4) — in code that hasn't changed since the last review.

.github/workflows/build-failure-analysis-command.md:471

• The job/artifact identity is made lossy before mapping: lowercasing, deleting all non-alphanumeric characters, and deduplicating collapses distinct timeline jobs such as Foo-Bar and Foo_Bar to foobar. The later artifact loop can then treat both artifacts as the same job, so if one is failed it may download the successful job's binlog too; preserve exact names and only use a normalized/prefix key when its identity is unique.
  .github/workflows/build-failure-analysis.md:407
• The job/artifact identity is made lossy before mapping: lowercasing, deleting all non-alphanumeric characters, and deduplicating collapses distinct timeline jobs such as Foo-Bar and Foo_Bar to foobar. The later artifact loop can then treat both artifacts as the same job, so if one is failed it may download the successful job's binlog too; preserve exact names and only use a normalized/prefix key when its identity is unique.
  .github/workflows/build-failure-analysis-command.md:154
• activation does not require binlog-found: the top-level if and generated lock gate on analysis-ready, while binlog-found only controls artifact upload/download. As written, this comment says the valid hlx-only path is disabled; update it to describe the actual gate.

This issue also appears on line 535 of the same file.
.github/workflows/build-failure-analysis.md:476

• The guard is now MAX_ZIP_BYTES=2147483648 (2 GB), but this explanation still says the per-artifact cap is 500 MB and that an ordinary build trips it. That is materially misleading for operators assessing disk/network exposure and for anyone deciding whether a skipped artifact should be expected; update the comment to describe the actual 2 GB limit and fallback behavior.

.github/workflows/build-failure-analysis-command.md:802

• This has the same failure mode as the automatic workflow: after analysis-ready=true, a transient artifact-service/upload failure fails fetch-binlog and suppresses the agent, rather than allowing the hlx-only path to run. Make the artifact handoff best-effort and let the agent proceed with no downloaded binlogs when the handoff is unavailable.

      - name: Upload analysis artifact

.github/workflows/build-failure-analysis-command.md:538

• The guard is now MAX_ZIP_BYTES=2147483648 (2 GB), but this explanation still says the per-artifact cap is 500 MB and that an ordinary build trips it. That is materially misleading for operators assessing disk/network exposure and for anyone deciding whether a skipped artifact should be expected; update the comment to describe the actual 2 GB limit and fallback behavior.

          # A 500 MB per-artifact cap is close enough to the size of a real
          # log artifact that an ordinary build trips it, and the job then
          # silently skips exactly the leg it exists to diagnose. Only one
          # archive is on disk at a time (each is deleted before the next

.github/workflows/build-failure-analysis-command.md:486

• The artifact-list request only consumes .value from one response. Azure DevOps paginates this endpoint via a continuation token, and this workflow explicitly expects roughly 150 Logs_Build_* artifacts; a failed leg on a later page will be missed and the agent may analyze incomplete evidence. Preserve and follow the continuation token (or otherwise prove the response is complete) before matching names.

          ado_get "artifact list" "${ADO_API}/build/builds/${BUILD_ID}/artifacts?api-version=7.1" || emit_none
          artifacts_json="${ADO_DOC}"
          mapfile -t all_names < <(printf '%s' "${artifacts_json}" | jq -r '.value // [] | map(select(.name | test("^Logs_Build_"))) | .[].name')

.github/workflows/build-failure-analysis-command.md:375

• This is another non-full-string numeric check: grep -q can accept the first all-digit line of a multi-line value. If the event/aw_context payload is malformed, PR_NUMBER can pass and then reach API paths and workflow-command logging. Use a Bash character-class check (or equivalent) that rejects any non-digit, including newlines.

          if ! printf '%s' "${PR_NUMBER}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis-command.md:411

• This API-response validation has the same per-line grep -qE '^[0-9]+$' flaw: a malformed JSON field containing a numeric first line can pass, and BUILD_ID is then interpolated into later ::warning:: output and ADO URLs. Make the check reject every non-digit character rather than matching any line.

          if ! printf '%s' "${BUILD_ID}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis-command.md:777

• The final gate only re-reads the PR head/merge SHA, so a newer runtime build can complete successfully for the same head while this command run is downloading/analyzing the older failed build; because this workflow sets cancel-in-progress: false, it will still post the stale failure analysis. Re-query the PR's latest runtime build and require the analyzed build to remain the newest failed build before allowing safe outputs, or otherwise invalidate this run when a newer build appears.

          LATEST_PR=$(gh api "repos/${GH_AW_REPO}/pulls/${PR_NUMBER}" 2>/dev/null)
          LATEST_HEAD=$(printf '%s' "${LATEST_PR}" | jq -r '.head.sha // empty')
          LATEST_MERGE=$(printf '%s' "${LATEST_PR}" | jq -r '.merge_commit_sha // empty')

.github/workflows/build-failure-analysis.md:739

• This upload is a hard step after the fetch script has emitted analysis-ready=true. A transient GitHub artifact-service/upload failure therefore marks fetch-binlog failed and prevents the downstream activation/agent jobs from running, so the documented hlx fallback is never reached even though binlogs are optional. Make the artifact handoff best-effort and have the agent continue with an empty binlog directory when upload/download fails.

        if: steps.fetch.outputs.binlog-found == 'true'

.github/workflows/build-failure-analysis.md:263

• This check is not actually full-string: grep applies ^...$ per line and -q succeeds if any line matches. A dispatch input such as 123\n456 therefore passes, after which BUILD_ID is used in ADO URLs and later ::warning:: output. Use a character-class check (or a null-delimited grep) that rejects every non-digit, including newlines.

          if ! printf '%s' "${BUILD_ID}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis.md:320

• The same per-line grep -qE '^[0-9]+$' issue applies here: a malformed/free-form PR input containing a newline after a numeric line can pass validation, then be interpolated into GitHub API paths and ::warning:: messages. Replace this with a full-string character-class check that rejects newlines and all other non-digits.

          if ! printf '%s' "${PR_NUMBER}" | grep -qE '^[0-9]+$'; then

.github/workflows/build-failure-analysis.md:422

• The artifact-list request only consumes .value from one response. Azure DevOps paginates this endpoint via a continuation token, and this workflow explicitly expects roughly 150 Logs_Build_* artifacts; a failed leg on a later page will be missed and the agent may analyze incomplete evidence. Preserve and follow the continuation token (or otherwise prove the response is complete) before matching names.

          ado_get "artifact list" "${ADO_API}/build/builds/${BUILD_ID}/artifacts?api-version=7.1" || emit_none
          artifacts_json="${ADO_DOC}"
          mapfile -t all_names < <(printf '%s' "${artifacts_json}" | jq -r '.value // [] | map(select(.name | test("^Logs_Build_"))) | .[].name')

.github/workflows/shared/build-failure-analysis-shared.md:80

• The workflow declares the same required safe-outputs.data schema for both add_comment and create_pull_request_review_comment, but this instruction requires the structured object only for the summary. The agent's inline suggestion calls therefore have no required data payload and can be rejected by safe-output validation (or lose the artifact identity); require the object on every inline output as well.

   - Post exactly one summary via `add_comment` with structured data
     `{"workflow_artifact":"build-failure-analysis","artifact_kind":"analysis"}`
     and any inline
     `suggestion` blocks via `create_pull_request_review_comment`. Both

• Files reviewed: 5/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Moderate issues affect artifact matching, extraction completeness, and download-budget enforcement.

Review details

Suppressed comments (9)

.github/workflows/build-failure-analysis-command.md:746

• The extraction loop reads find output as newline-delimited text. Archive entry names are PR-controlled and may contain newlines, so one *.binlog path can be split into multiple records; the fragments fail the [ -f ] check and are silently ignored, while any other files from the same artifact still make leg_staged > 0 and the leg is accepted. This contradicts the all-or-nothing comment and can omit evidence from a selected failed leg. Use NUL-delimited find -print0 with read -d '' (or otherwise reject malformed paths) so an unrepresentable entry fails the leg rather than being dropped.

            done < <(find "${AX_DIR}" -type f -name '*.binlog')

.github/workflows/build-failure-analysis-command.md:154

• This comment still says activation additionally requires binlog-found == 'true', but the workflow's top-level gate is analysis-ready == 'true' and the intended hlx-only path sets analysis-ready even when no binlog was found. The stale description can lead future edits to disable the no-binlog fallback; update it to say activation requires the verified failed build/revision, while binlog retrieval is optional.

    # and `activation` additionally requires `binlog-found == 'true'`.

.github/workflows/build-failure-analysis-command.md:645

• The cumulative compressed-download budget only adds the final size left in ZIP_TMP, but curl --retry 3 can download up to the cap on each failed attempt before truncating/replacing that file. Thus a nominal 3 GB total can still pull roughly four times that amount from Azure DevOps (and a hostile/unstable endpoint can consume the network/time budget without being charged). If this is intended as a network-egress ceiling, account for each attempt or disable retries/use a transfer mechanism with a true cumulative byte limit.

              timeout "${TIME_LEFT}" curl -sSL --fail --retry 3 --retry-delay 2 --connect-timeout 15 --max-time "${ATTEMPT_SECONDS}" --retry-max-time "${TIME_LEFT}" -o "${ZIP_TMP}" "${url}"
            ) 2>/dev/null
            curl_rc=$?
            ZIP_BYTES=$(stat -c%s "${ZIP_TMP}" 2>/dev/null || echo 0)
            # Charge the budget with the bytes retained on disk, including those of an
            # artifact about to be skipped. This is a disk and extraction budget, not a
            # meter of network egress: `-o` truncates before each retry, so failed
            # attempts are not counted here. What bounds those is FETCH_DEADLINE via
            # the `timeout` wrapper, plus `ulimit -f`, which caps every individual
            # attempt at ZIP_CAP.
            TOTAL_ZIP_BYTES=$((TOTAL_ZIP_BYTES + ZIP_BYTES))

.github/workflows/build-failure-analysis-command.md:514

• The artifact-to-job association is not actually exact here: when no exact normalized name matches, this accepts any artifact key that is a prefix of exactly one timeline job. For example, Logs_Build_NativeAOT would be mapped to the sole NativeAOT_Libraries job even though the artifact name does not identify that job. This can select a successful/non-corresponding artifact and contradicts the workflow's fail-closed “exact failed/canceled-job artifact” guarantee; use an authoritative artifact/job association or skip artifacts without an exact match.

              if [ -z "${mapped_job_key}" ]; then
                prefix_matches=0
                for job_key in "${all_job_keys[@]}"; do
                  if [[ "${job_key}" == "${artifact_key}"* ]]; then
                    mapped_job_key="${job_key}"
                    prefix_matches=$((prefix_matches + 1))
                  fi
                done
                [ "${prefix_matches}" -eq 1 ] || mapped_job_key=""

.github/workflows/build-failure-analysis-command.md:471

• The normalization here is lossy (tr -cd '[:alnum:]') and all_job_keys has already deduplicated the results. Distinct timeline jobs such as Foo-Bar and FooBar therefore collapse to one key; an artifact for the other job can be treated as an exact match and selected even when that job succeeded. Preserve the original job names (and reject normalized collisions) before using prefix/exact matching, otherwise the workflow can analyze the wrong leg instead of only failed/canceled jobs.

          mapfile -t failed_job_keys < <(
            printf '%s' "${timeline_json}" |
              jq -r '.records // [] | map(select(.type == "Job" and (.result == "failed" or .result == "canceled"))) | .[].name' |
              while IFS= read -r job_name; do
                printf '%s' "${job_name}" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]'
                printf '\n'
              done |
              awk 'NF && !seen[$0]++'

.github/workflows/build-failure-analysis.md:682

• The extraction loop reads find output as newline-delimited text. Archive entry names are PR-controlled and may contain newlines, so one *.binlog path can be split into multiple records; the fragments fail the [ -f ] check and are silently ignored, while any other files from the same artifact still make leg_staged > 0 and the leg is accepted. This contradicts the all-or-nothing comment and can omit evidence from a selected failed leg. Use NUL-delimited find -print0 with read -d '' (or otherwise reject malformed paths) so an unrepresentable entry fails the leg rather than being dropped.

            done < <(find "${AX_DIR}" -type f -name '*.binlog')

.github/workflows/build-failure-analysis.md:450

• The artifact-to-job association is not actually exact here: when no exact normalized name matches, this accepts any artifact key that is a prefix of exactly one timeline job. For example, Logs_Build_NativeAOT would be mapped to the sole NativeAOT_Libraries job even though the artifact name does not identify that job. This can select a successful/non-corresponding artifact and contradicts the workflow's fail-closed “exact failed/canceled-job artifact” guarantee; use an authoritative artifact/job association or skip artifacts without an exact match.

              if [ -z "${mapped_job_key}" ]; then
                prefix_matches=0
                for job_key in "${all_job_keys[@]}"; do
                  if [[ "${job_key}" == "${artifact_key}"* ]]; then
                    mapped_job_key="${job_key}"
                    prefix_matches=$((prefix_matches + 1))
                  fi
                done
                [ "${prefix_matches}" -eq 1 ] || mapped_job_key=""

.github/workflows/build-failure-analysis.md:581

• The cumulative compressed-download budget only adds the final size left in ZIP_TMP, but curl --retry 3 can download up to the cap on each failed attempt before truncating/replacing that file. Thus a nominal 3 GB total can still pull roughly four times that amount from Azure DevOps (and a hostile/unstable endpoint can consume the network/time budget without being charged). If this is intended as a network-egress ceiling, account for each attempt or disable retries/use a transfer mechanism with a true cumulative byte limit.

              timeout "${TIME_LEFT}" curl -sSL --fail --retry 3 --retry-delay 2 --connect-timeout 15 --max-time "${ATTEMPT_SECONDS}" --retry-max-time "${TIME_LEFT}" -o "${ZIP_TMP}" "${url}"
            ) 2>/dev/null
            curl_rc=$?
            ZIP_BYTES=$(stat -c%s "${ZIP_TMP}" 2>/dev/null || echo 0)
            # Charge the budget with the bytes retained on disk, including those of an
            # artifact about to be skipped. This is a disk and extraction budget, not a
            # meter of network egress: `-o` truncates before each retry, so failed
            # attempts are not counted here. What bounds those is FETCH_DEADLINE via
            # the `timeout` wrapper, plus `ulimit -f`, which caps every individual
            # attempt at ZIP_CAP.
            TOTAL_ZIP_BYTES=$((TOTAL_ZIP_BYTES + ZIP_BYTES))

.github/workflows/build-failure-analysis.md:407

• The normalization here is lossy (tr -cd '[:alnum:]') and all_job_keys has already deduplicated the results. Distinct timeline jobs such as Foo-Bar and FooBar therefore collapse to one key; an artifact for the other job can be treated as an exact match and selected even when that job succeeded. Preserve the original job names (and reject normalized collisions) before using prefix/exact matching, otherwise the workflow can analyze the wrong leg instead of only failed/canceled jobs.

          mapfile -t failed_job_keys < <(
            printf '%s' "${timeline_json}" |
              jq -r '.records // [] | map(select(.type == "Job" and (.result == "failed" or .result == "canceled"))) | .[].name' |
              while IFS= read -r job_name; do
                printf '%s' "${job_name}" | tr '[:upper:]' '[:lower:]' | tr -cd '[:alnum:]'
                printf '\n'
              done |
              awk 'NF && !seen[$0]++'

• Files reviewed: 5/7 changed files
• Comments generated: 0 new
• Review effort level: Lite

[YuliiaKovalova]

Published the remaining fixes through cba969a, preserving the intervening merge from main. The 17 existing review threads were already resolved when refreshed; the three original discussions now have corrective follow-ups describing the final behavior.

For the suppressed findings in review 5224911875 and review 5234721767:

Area	Resolution
Artifact/job ambiguity	Removed normalized-name and prefix heuristics. Artifacts now join to exactly one failed/canceled timeline job by the documented BuildArtifact.source producing-job ID. Missing/unknown IDs use hlx fallback.
Extraction completeness and numeric guards	NUL-delimited staging preserves newline-containing paths. Numeric identifiers and sizes are validated as whole strings. Current source-driven regressions cover both.
Optional artifact handoff	Upload/download failures warn rather than block hlx analysis; availability reflects actual action success, and failed downloads discard partial binlogs.
Stale output and contributor reads	Agent-side checks use the supported MCP head field. The deterministic final job checks REST head/merge plus the latest failed build. MCP reads are repository-scoped with integrity filtering suitable for external PRs.
Command activation and idempotency	Exact raw activation, non-cancelling queued requests, exact request-URL deduplication, and deterministic structured metadata; framework failure/status comments do not masquerade as completed analyses.
Pinned binlog tools	Fork E2E exposed that binlog_* matched no tools in the pinned gateway. Both workflows now list the 35 advertised binlog tools explicitly, excluding management tools.

No artifact-pagination shim was added: the documented Build Artifacts List contract exposes no continuation parameter. Missing outer data does not itself cause schema rejection, but real E2E showed it broke persisted analysis recognition; that actual defect is fixed. Compressed-size limits govern retained data/extraction, while retries share a deadline; comments now avoid claiming cumulative network-egress accounting. The stale activation-gate description was corrected without disabling hlx-only analysis.

Both workflows pass pinned strict compilation and all 13 regression tests. Current-source automatic replay, command plus forced metadata omission, and same-URL deduplication demonstrate real binary diagnostics, publication, metadata repair, and duplicate suppression.

Proof limitations are explicit in the updated PR description: automatic delivery was replayed; one current-source command attempt failed output-quality acceptance and required a guided retry; complete successful reads of every supplied binlog and consistently good bare-command output are not established. Generated suggestions were not compiled. The fork default branch was restored, and no personal credential was uploaded.

This review follow-up was generated by GitHub Copilot.

[Copilot]

🔵 Needs a closer look

Unresolved archive size-enforcement and critical extraction-path test-coverage findings remain.

Review details

Suppressed comments (3)

.github/workflows/build-failure-analysis-command.md:613

• ulimit -f is configured in 1024-byte blocks, but rounding (ZIP_CAP + 1023) / 1024 up means the kernel permits a file larger than ZIP_CAP (by up to 1023 bytes). Because the post-download check only rejects >= ZIP_CAP, a no-Content-Length response can exceed both the per-artifact and remaining cumulative compressed-byte budgets. Round down (or add an exact-size enforcement loop) so the backstop cannot exceed the advertised cap.

[!NOTE] This review was created by GitHub Copilot.

              ulimit -f $(( (ZIP_CAP + 1023) / 1024 )) || exit 1

.github/workflows/build-failure-analysis.md:551

• ulimit -f is configured in 1024-byte blocks, but rounding (ZIP_CAP + 1023) / 1024 up means the kernel permits a file larger than ZIP_CAP (by up to 1023 bytes). Because the post-download check only rejects >= ZIP_CAP, a no-Content-Length response can exceed both the per-artifact and remaining cumulative compressed-byte budgets. Round down (or add an exact-size enforcement loop) so the backstop cannot exceed the advertised cap.

[!NOTE] This review was created by GitHub Copilot.

              ulimit -f $(( (ZIP_CAP + 1023) / 1024 )) || exit 1

.github/workflows/build-failure-analysis.md:468

• The new archive-handling path has no test that exercises the real curl/ulimit/unzip flow: the existing staging test mocks find, [ and cp, so it never verifies compressed-size accounting, uncompressed-size parsing, traversal rejection, extraction failure handling, or the cumulative/time budget exits. These guards are security- and reliability-critical and are duplicated in the command workflow; please add a fixture-driven test (or a focused script test) that runs the actual loop against small ZIPs and covers both accepted and rejected archives, including budget exhaustion.

          MAX_ZIP_BYTES=2147483648      # 2 GB compressed per artifact
          MAX_UNZIP_BYTES=2147483648    # 2 GB uncompressed per artifact
          MAX_TOTAL_BYTES=4294967296    # 4 GB uncompressed across all artifacts
          # Raising the per-artifact cap would otherwise raise the worst-case
          # number of bytes pulled over the network by the same factor, since
          # nothing else bounds the sum across artifacts. Cap the total
          # download too, and charge it *before* each transfer (see ZIP_CAP
          # below) rather than after, so the last artifact can't start just
          # under the limit and still pull a full MAX_ZIP_BYTES.

• Files reviewed: 7/9 changed files
• Comments generated: 0 new
• Review effort level: Lite

[YuliiaKovalova]

@PureWeen & @jeffhandley could you please review it again? sorry for the long break, I was OOF.

[PureWeen]

Holistic Review

Motivation: Automating build-failure analysis is valuable, and the new revision addresses the three workflow activation and command-routing issues from my earlier review.

Approach: PR identity is now validated against ADO metadata and the current PR head, comment runs no longer cancel each other, and the source command predicate matches the generated activation logic. One safe-output retry issue remains.

Summary: Needs Changes at cba969a2. The previous three findings are fixed. The persisted-summary dedup below can permanently omit inline findings or duplicate them after a partial safe-output failure.

Detailed Findings

⚠️ Warning — The summary comment is not an atomic completion marker

.github/workflows/build-failure-analysis-command.md:369-394 treats an existing summary carrying the request metadata/footer as proof that the entire safe-output batch completed. With pinned gh-aw v0.86.2, however, add_comment is written immediately, inline review comments are buffered, and submitReview() occurs only after message processing. Handler/finalization errors are accumulated and fail the job after earlier writes.

If the summary succeeds and final review submission fails, an edit or redelivery of the same command URL finds the summary and skips permanently, leaving the inline feedback missing. If the summary fails after buffered review submission succeeds, retry can duplicate the inline comments.

Please key deduplication on a confirmed successful safe-output run or a completion marker written only after review finalization, rather than on summary presence alone. Add a partial-failure/retry test; the current E2E covers only the all-success publication path.

Focused source-driven validation passed all 13 regression tests. I did not inject a live GitHub partial-output failure or assess CI/merge readiness.

Note

This review was generated by GitHub Copilot.

[YuliiaKovalova]

Addressed the retry issue from #132609 (review) in 3e411a5; the equivalent fix is also pushed to dotnet/roslyn#85046 (17e5bd12064).

Command deduplication now checks the exact command-ID run name and requires a completed attempt with a successful safe_outputs job and Process Safe Outputs step. Summary presence is no longer completion evidence. Partial retries filter workflow-generated fingerprints on published summaries and submitted review comments/bodies: one summary per request/revision, with inline identity including the revision, anchor, and full finding body. Distinct finding bodies remain distinct; pending reviews do not suppress retries. API failures and incomplete run searches fail closed, and only the fetch job gains actions: read.

Validation: reproduced the old premature-skip regression before the fix; all 22 Runtime source-driven tests pass, including summary-success/review-failure, summary-failure/review-success, failure after durable writes, and the 1,000-run boundary. Both workflow locks compile with pinned gh-aw v0.86.2 using --strict --validate --schedule-seed dotnet/runtime. Roslyn has the same 8 publication regressions passing and both workflows compile with its repository seed. These are mocked API fault injections, not live GitHub partial-output failure runs; no Azure build was rerun.

Note

This update was generated by GitHub Copilot.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings remain in workflow completion, archive extraction, timeout/entry bounds, and safe-output publication.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3

Open (3)

• Require verified agent completion before suppressing retries · New
• Reject symlink and device entries in archives · New
• Reject symlink and device entries in archives · New

[YuliiaKovalova]

Confirmed against the current head. A mocked completed attempt containing a failed agent CLI step and a successful safe_outputs / Process Safe Outputs pair returns completed=true from the production predicate, even with an empty output placeholder. The generated lock writes the placeholder under always(), and pinned gh-aw treats an empty output list as successful processing.

The same completion predicate is present in Roslyn PR dotnet/roslyn#85046, so the follow-up must cover both repositories. Successful agent execution/output ingestion must also be established without rejecting a deliberate noop. This is still unaddressed, so I am leaving this thread open rather than marking the prior publication-retry fix as sufficient.

Note

This assessment was generated by GitHub Copilot.