Onboard build failure analysis workflow

.github/aw/actions-lock.json

@@ -50,5 +50,17 @@
       "version": "v0.86.2",
       "sha": "6aab9e5b5c91c615506061f09bedd81a23babe3c"
     }
+  },
+  "containers": {
+    "ghcr.io/lewing/helix.mcp:v0.8.0": {
+      "image": "ghcr.io/lewing/helix.mcp:v0.8.0",
+      "digest": "sha256:3c0a16544d922f17ebb477df4d4b0bd546815eb6b1200f4fd8291809011fc20f",
+      "pinned_image": "ghcr.io/lewing/helix.mcp:v0.8.0@sha256:3c0a16544d922f17ebb477df4d4b0bd546815eb6b1200f4fd8291809011fc20f"
+    },
+    "mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-binlog-mcp-amd64": {
+      "image": "mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-binlog-mcp-amd64",
+      "digest": "sha256:253736e28e0230269dfcdb70f5027da47e2e45e8526d15d6485ca08b2c2f1638",
+      "pinned_image": "mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-binlog-mcp-amd64@sha256:253736e28e0230269dfcdb70f5027da47e2e45e8526d15d6485ca08b2c2f1638"
+    }
   }
 }

.github/workflows/README.md

@@ -22,3 +22,33 @@ jobs:
 Refer to GitHub's [Workflows in forked repositories](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#workflows-in-forked-repositories) and [pull_request_target](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#pull_request_target) documentation for more information.
 
 Agentic workflow safe outputs sanitize posted bodies and [remove agent-provided HTML/XML comments](https://github.github.com/gh-aw/reference/safe-outputs/#text-sanitization-allowed-domains-allowed-github-references). Do not use HTML comments as machine-readable markers or persisted state. Prefer schema-validated `safe-outputs.data` when the output supports it and another fenced JSON block is compatible with downstream consumers; otherwise use stable visible fields. See [the repository agentic-workflow guidance](../agents/agentic-workflows.agent.md#repository-specific-requirements-safe-output-data) for authoring and migration requirements.
+
+## Build-failure analysis regression checks
+
+With Python 3, PyYAML, Bash, jq, and Node.js installed, run
+`python -m unittest discover -s .github/workflows/tests -p "test_*.py"`.
+The tests execute the workflow's own Bash, jq, and JavaScript with mocked API responses;
+they do not download artifacts or post to GitHub. On Windows they use Git Bash
+by default; `BFA_BASH` can select another Bash executable.
+
+Artifacts are associated with failed/canceled timeline jobs using Azure DevOps'
+`BuildArtifact.source` job ID, not normalized names or prefixes. Missing or
+unknown source IDs leave those jobs to the mandatory hlx task-log analysis.
+
+Command runs use the command-comment ID in their run name. A repeated request
+is skipped only after that workflow's `safe_outputs` job and its processing step
+have succeeded in a completed run attempt. A posted summary alone is not
+completion evidence. API failures and histories exceeding GitHub's 1,000-run
+search limit stop the check rather than authorizing duplicate publication.
+
+Partial retries retain one summary per request/revision and suppress identical
+inline findings using visible, workflow-generated fingerprints of the request,
+revision, anchor, and body. Submitted review bodies are checked too, including
+gh-aw's unanchored-comment fallback; pending reviews are not publication evidence.
+Different requests, revisions, or finding bodies remain distinct. Regression
+tests inject summary and review-submission failures independently and verify
+that retries publish only the missing outputs; they do not inject live API faults.
+
+Regenerate both locks with
+`gh aw compile build-failure-analysis build-failure-analysis-command --strict --validate --schedule-seed dotnet/runtime`
+before running the checks against a changed activation configuration.

.github/workflows/shared/build-failure-analysis-shared.md

@@ -0,0 +1,220 @@
+---
+# Shared body for the build-failure-analysis workflows.
+#
+# Imported by build-failure-analysis.md (check_run + workflow_dispatch
+# triggers) and build-failure-analysis-command.md (slash command). Keeps the
+# prompt that drives the build-failure analysis in one place. Per-trigger
+# wiring (steps, env, mcp-servers, permissions) lives in each caller because
+# gh-aw merges those fields from imports but each main workflow must still
+# re-declare its top-level permissions.
+
+description: "Shared body for build-failure-analysis workflows"
+
+# Callers must not override steps: gh-aw replaces that array rather than merging
+# it. Keep metadata materialization and the final write guard together here.
+safe-outputs:
+  steps:
+    - name: Ensure build-analysis output metadata
+      if: steps.download-agent-output.outcome == 'success'
+      uses: actions/github-script@v9.0.0
+      env:
+        GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }}
+      with:
+        script: |
+          const fs = require("node:fs");
+          const outputPath = process.env.GH_AW_AGENT_OUTPUT;
+          const output = JSON.parse(fs.readFileSync(outputPath, "utf8"));
+          if (!Array.isArray(output.items)) {
+            throw new Error("Build-analysis output must contain an items array.");
+          }
+          for (const item of output.items) {
+            if (item.type !== "add_comment" && item.type !== "create_pull_request_review_comment") {
+              continue;
+            }
+            if (typeof item.body !== "string") {
+              throw new Error("Build-analysis comments must have a string body.");
+            }
+            if (item.data === undefined) {
+              item.data = { workflow_artifact: "build-failure-analysis", artifact_kind: "analysis" };
+            } else if (item.data === null || Array.isArray(item.data) ||
+                       Object.keys(item.data).length !== 2 ||
+                       item.data.workflow_artifact !== "build-failure-analysis" ||
+                       item.data.artifact_kind !== "analysis") {
+              throw new Error("Build-analysis comment metadata does not match the workflow schema.");
+            }
+            // MCP normalizes supplied data into the body, but data is optional
+            // at that boundary. Materialize the same block when it was omitted.
+            const block = "Structured data:\n```json\n" + JSON.stringify(item.data, null, 2) + "\n```";
+            if (!item.body.includes(block)) {
+              item.body += "\n\n" + block;
+            }
+          }
+          fs.writeFileSync(outputPath, JSON.stringify(output));
+    - name: Prepare retry-safe command outputs
+      if: github.event_name == 'issue_comment' && steps.download-agent-output.outcome == 'success'
+      uses: actions/github-script@v9.0.0
+      env:
+        GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }}
+        EXPECTED_HEAD: ${{ needs.fetch-binlog.outputs.pr-head-sha }}
+      with:
+        script: |
+          const fs = require("node:fs");
+          const { createHash } = require("node:crypto");
+          const request = context.payload.comment.id;
+          const pullNumber = context.payload.issue.number;
+          const head = process.env.EXPECTED_HEAD;
+          if (!Number.isSafeInteger(request) || !/^[a-f0-9]{40}$/.test(head)) {
+            throw new Error("Missing verified command or revision identity.");
+          }
+          const outputPath = process.env.GH_AW_AGENT_OUTPUT;
+          const output = JSON.parse(fs.readFileSync(outputPath, "utf8"));
+          if (!Array.isArray(output.items)) throw new Error("Expected an output items array.");
+          const comments = await github.paginate(github.rest.issues.listComments, {
+            ...context.repo, issue_number: pullNumber, per_page: 100,
+          });
+          const reviews = await github.paginate(github.rest.pulls.listReviews, {
+            ...context.repo, pull_number: pullNumber, per_page: 100,
+          });
+          const inline = await github.paginate(github.rest.pulls.listReviewComments, {
+            ...context.repo, pull_number: pullNumber, per_page: 100,
+          });
+          const isBot = item => item.user?.login === "github-actions[bot]" && item.user?.type === "Bot";
+          const submitted = reviews.filter(review => isBot(review) && review.state !== "PENDING" && review.submitted_at);
+          const reviewIds = new Set(submitted.map(review => review.id));
+          // Include review bodies: gh-aw moves unanchorable findings there.
+          const published = [...comments.filter(isBot), ...submitted,
+            ...inline.filter(item => isBot(item) && reviewIds.has(item.pull_request_review_id))];
+          const markers = new Set(published.flatMap(item =>
+            [...(item.body || "").matchAll(/^Build-analysis output: `(\d+:[a-f0-9]{64})`$/gm)].map(match => match[1])));
+          output.items = output.items.filter(item => {
+            if (item.type !== "add_comment" && item.type !== "create_pull_request_review_comment") return true;
+            if (typeof item.body !== "string") throw new Error("Expected a comment body.");
+            const body = item.body.replace(/^Build-analysis output: `\d+:[a-f0-9]{64}`\n\n/, "").replace(/\r\n/g, "\n").trim();
+            // There is one summary per request/revision. Inline identity also
+            // includes the full finding and anchor, so distinct findings survive.
+            const identity = item.type === "add_comment" ? [request, head, item.type] :
+              [request, head, item.type, item.path, Number(item.line), item.side || "RIGHT",
+                item.start_line ? Number(item.start_line) : null, body];
+            const key = `${request}:${createHash("sha256").update(JSON.stringify(identity)).digest("hex")}`;
+            if (markers.has(key)) {
+              core.info(`Skipping previously published ${item.type} (${key}).`);
+              return false;
+            }
+            markers.add(key);
+            item.body = `Build-analysis output: \`${key}\`\n\n${body}`;
+            return true;
+          });
+          fs.writeFileSync(outputPath, JSON.stringify(output));
+    - name: Revalidate PR revision before applying queued outputs
+      shell: bash
+      env:
+        GH_TOKEN: ${{ github.token }}
+        GH_AW_REPO: ${{ github.repository }}
+        PR_NUMBER: ${{ needs.fetch-binlog.outputs.pr-number }}
+        EXPECTED_HEAD: ${{ needs.fetch-binlog.outputs.pr-head-sha }}
+        EXPECTED_MERGE: ${{ needs.fetch-binlog.outputs.pr-merge-sha }}
+        BUILD_ID: ${{ needs.fetch-binlog.outputs.ado-build-id }}
+        ADO_API: "https://dev.azure.com/dnceng-public/public/_apis"
+        ADO_BUILD_DEFINITION_ID: "129"
+      run: |
+        set -euo pipefail
+        if [[ ! "${PR_NUMBER}" =~ ^[0-9]+$ || ! "${BUILD_ID}" =~ ^[0-9]+$ ]]; then
+          echo "::error::Missing or invalid verified PR/build identity before applying outputs."
+          exit 1
+        fi
+        # A rerun can succeed without changing either commit. Revalidate the
+        # latest build as well as the revisions before publishing old failures.
+        latest_build="${RUNNER_TEMP}/build-failure-analysis-latest-build.json"
+        trap 'rm -f "${latest_build}"' EXIT
+        if ! timeout 60 curl -sSL --fail --retry 3 --connect-timeout 10 --max-time 20 --retry-max-time 40 \
+             -o "${latest_build}" \
+             "${ADO_API}/build/builds?definitions=${ADO_BUILD_DEFINITION_ID}&branchName=refs/pull/${PR_NUMBER}/merge&queryOrder=queueTimeDescending&\$top=1&api-version=7.1" ||
+           ! jq -e --arg id "${BUILD_ID}" \
+             '.value[0] | (.id | tostring) == $id and .status == "completed" and .result == "failed"' \
+             "${latest_build}" >/dev/null; then
+          echo "::error::Analyzed build is no longer the latest completed failed runtime build, or could not be verified; refusing stale outputs."
+          exit 1
+        fi
+        if [ -z "${EXPECTED_HEAD}" ] || [ -z "${EXPECTED_MERGE}" ] ||
+           ! gh api "repos/${GH_AW_REPO}/pulls/${PR_NUMBER}" |
+             jq -e --arg head "${EXPECTED_HEAD}" --arg merge "${EXPECTED_MERGE}" \
+               '.head.sha == $head and .merge_commit_sha == $merge' >/dev/null; then
+          echo "::error::PR #${PR_NUMBER} moved or could not be verified before applying queued build-analysis outputs."
+          exit 1
+        fi
+---
+
+# Build Failure Analyst
+
+You are the **build-failure analyst**. Analyze the binary logs of the Azure
+DevOps build that just failed, plus its failed compile-task logs where Runtime
+did not publish a matching binlog, and produce a PR review using the
+safe-output tools (a later `safe_outputs` job performs the actual GitHub write).
+Do **not** try to spawn a sub-agent: the `task` tool is intentionally not
+available here. Work directly with the tools you do have: `binlog-mcp` to
+read the logs, the `github` tools to read PR/repo context (the GitHub MCP
+server is **read-only** here), the `safeoutputs` tools (`add_comment`,
+`create_pull_request_review_comment`, `noop`) to post results, and a small set
+of read-only `shell` commands (including `cat`).
+
+## Instructions
+
+1. Read the agent-context environment variables: `GH_AW_BUILD_OUTCOME`,
+   `GH_AW_BINLOG_LIST`, `GH_AW_BINLOG_DIR`, `GH_AW_BINLOG_PATH`,
+   `GH_AW_BINLOG_HOST_PATH`, `GH_AW_PR_NUMBER`, `GH_AW_PR_HEAD_SHA`,
+   `GH_AW_PR_MERGE_SHA`, `GH_AW_WORKSPACE`.
+
+2. If `GH_AW_BUILD_OUTCOME == 'success'`, the build did not actually fail —
+   there is nothing to analyze. Call `noop` with the message
+   `"Build succeeded — no analysis required."` and stop.
+
+3. Load your detailed playbook: `cat .github/agents/build-failure-analyst.agent.md`
+   (it is checked out with the repository config). Follow that methodology —
+   root-cause grouping, source-context reading via the GitHub API at
+   `GH_AW_PR_HEAD_SHA`, comment/suggestion formatting, and defensive behavior.
+   In summary:
+   - Start with `azdo_timeline` from the `hlx` MCP server for
+     `GH_AW_BINLOG_HOST_PATH` using
+     `filter: "failed"` to inventory **every** failed/canceled job and task.
+     Treat only compile/build/configure/link tasks as build evidence; Helix,
+     test execution, publishing, and infrastructure failures remain out of
+     scope. For each failed compile task that is not explained by a retrieved
+     binlog or by complete build diagnostics in its timeline `issues`, use
+     `azdo_search_log` from `hlx` against that task's `logId` with bounded
+     searches for compiler/MSBuild/native-build failure signatures. This is
+     required even when some binlogs were retrieved: Runtime job display names
+     and artifact names are not one-to-one, and some compile jobs publish no
+     `Logs_Build_*` artifact.
+   - Iterate **every** path in `GH_AW_BINLOG_LIST` when the list is non-empty
+     (newline-separated in-container binlog paths from failed/canceled build
+     jobs, under `GH_AW_BINLOG_DIR` = `/data/binlogs`) and query the
+     `binlog-mcp` MCP server (`binlog_errors`, `binlog_overview`,
+     `binlog_warnings`, …) with `binlog_file` set to each leg's path — a
+     failure usually surfaces in only one leg, so do not analyse just the
+     first. `binlog_errors`,
+     `binlog_overview`, `binlog_warnings`, … are **MCP tools** provided by the
+     `binlog-mcp` server: prefer calling them **directly as MCP tools** (with a
+     `binlog_file` argument). A CLI wrapper is also mounted and allowlisted, so
+     you may alternatively run `binlog-mcp <tool> --binlog_file <path>` via the
+     shell.
+   - If no binlog shows errors or failed-target/process evidence **and** the
+     bounded hlx task-log checks show no compile/build failure, the build work
+     compiled cleanly — the pipeline failure is then a **non-build**
+     (test/Helix/publishing/infrastructure) failure, which is **out of scope**.
+     Only make that clean-build conclusion when all required binlog and hlx
+     queries succeeded. Then **post nothing**: call `noop` with a short reason
+     and stop. Do **not** post a summary comment and do **not** invent fixes.
+     If a required query fails and the gap prevents classification, post one
+     incomplete-analysis summary with the Azure DevOps build link and no fix
+     claim or inline suggestion.
+   - Post exactly one summary via `add_comment` with structured data
+     `{"workflow_artifact":"build-failure-analysis","artifact_kind":"analysis"}`
+     and any inline
+     `suggestion` blocks via `create_pull_request_review_comment`. Both
+     workflows bind safe outputs deterministically to `GH_AW_PR_NUMBER`; do
+     not attempt to choose or override the target in a safe-output call.
+   - `submit_pull_request_review` is **not** a safe output for this workflow;
+     inline comments stand alone.
+
+4. When you have posted the analysis for a genuine build failure (or called
+   `noop` for a clean-compile / non-build failure), stop.