debug: Noop smoke pipeline fails to pull non-existent ghcr.io/github/gh-aw-mcpg:v0.3.7 (manifest unknown)

[jamesadevine]

Summary

The Daily safe-output smoke noop pipeline (Azure DevOps definition Noop smoke, id 2506, in msazuresphere/AgentPlayground) is failing in the Agent job at the Pre-pull AWF and MCPG container images (v0.25.44) step. The compiled lock file generated by ado-aw v0.30.1 (and also v0.30.2 on main) pins the MCP Gateway image to ghcr.io/github/gh-aw-mcpg:v0.3.7, a tag that does not exist in the registry. Docker returns Error response from daemon: manifest unknown and the job exits.

This has caused every daily smoke run since at least 16 May to fail.

Run under investigation

• Pipeline: Noop smoke (definition id 2506)
• Build: 607153 — Daily safe-output smoke noop-607153
• URL: https://dev.azure.com/msazuresphere/AgentPlayground/_build/results?buildId=607153&view=logs&j=6c181f5c-fa64-5337-60e0-f5bfb300ad55&t=7a6f47d6-c3f0-5ed7-7b53-50831ee9d6a7
• Branch / commit: refs/heads/main @ 9597f35c (source repo: githubnext/ado-aw)
• Source pipeline file: tests/safe-outputs/noop.lock.yml
• Compiler version recorded in lock header: @ado-aw source="tests/safe-outputs/noop.md" version=0.30.1
• Compiled with embedded AWF version: v0.25.44
• Started: 2026-05-18 13:48:08 UTC — Finished: 2026-05-18 13:50:07 UTC (≈2 min)
• Result: failed

Recent run history (definition 2506) shows uninterrupted failures:

Build	Result	Started (UTC)
607153	failed	2026-05-18 13:48:08
607067	failed	2026-05-18 03:36:54
606890	failed	2026-05-17 03:36:53
606819	failed	2026-05-16 21:27:24
606813	failed	2026-05-16 20:28:59
606812	failed	2026-05-16 20:14:19

Failure classification

Stage / Job	Step	Result
__default Stage	—	failed
Agent Phase / Job	—	failed
Pre-pull AWF and MCPG container images (v0.25.44) (order 16, log 19)	Bash exited with code 1	failed (root cause)

Category per debug rubric: Agent — download/setup → AWF Container Startup, specifically an MCPG image pull failure caused by the compiler emitting a non-existent image tag in the generated YAML.

Root cause

tests/safe-outputs/noop.lock.yml contains this step (verbatim from the generated file on main and on v0.30.2):

docker pull ghcr.io/github/gh-aw-firewall/squid:0.25.44
docker pull ghcr.io/github/gh-aw-firewall/agent:0.25.44
docker tag ghcr.io/github/gh-aw-firewall/squid:0.25.44 ghcr.io/github/gh-aw-firewall/squid:latest
docker tag ghcr.io/github/gh-aw-firewall/agent:0.25.44 ghcr.io/github/gh-aw-firewall/agent:latest
docker pull ghcr.io/github/gh-aw-mcpg:v0.3.7

The squid (0.25.44) and agent (0.25.44) images pull successfully. The final pull of ghcr.io/github/gh-aw-mcpg:v0.3.7 fails:

2026-05-18T13:49:43.1698657Z ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.3763131Z Error response from daemon: manifest unknown
2026-05-18T13:49:43.3838247Z ##[error]Bash exited with code '1'.

Querying ghcr.io/v2/github/gh-aw-mcpg directly (anonymous pull token):

• GET /v2/github/gh-aw-mcpg/manifests/v0.3.7 → 404 Not Found
• GET /v2/github/gh-aw-mcpg/manifests/0.3.7 → 404 Not Found

The published tag list for github/gh-aw-mcpg jumps from v0.2.16 directly to nothing higher — there is no v0.3.x series. Tags currently present include latest, v0.0.94 … v0.2.16 and matching commit-SHA tags. No v0.3.7 was ever pushed.

The lock.yml on main (compiler v0.30.1) and on tag v0.30.2 (released 2026-05-18 13:33 UTC, ~15 min before this build queued) both still emit ghcr.io/github/gh-aw-mcpg:v0.3.7, so upgrading the compiler to the latest release does not resolve this.

This is a compiler bug: ado-aw is generating a docker pull line that pins MCPG to a tag that does not exist in the published registry. Either the compiler's default MCPG tag constant needs to be moved back to a published tag (e.g. v0.2.16) or the missing gh-aw-mcpg:v0.3.7 image needs to be published.

Evidence

Tail of Pre-pull AWF and MCPG container images (v0.25.44) log:

2026-05-18T13:49:18.6247280Z Digest: sha256:bc13486f73cc9b8327e276839f936ecf2492cf86a9bee9114d2b018bdfc54782
2026-05-18T13:49:18.6264430Z Status: Downloaded newer image for ghcr.io/github/gh-aw-firewall/squid:0.25.44
...
2026-05-18T13:49:43.1679145Z Digest: sha256:71511dd944cd13a4d980a57bcf7a071f8662cb482686c58db294d9903e0960e4
2026-05-18T13:49:43.1691265Z Status: Downloaded newer image for ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.1698657Z ghcr.io/github/gh-aw-firewall/agent:0.25.44
2026-05-18T13:49:43.3763131Z Error response from daemon: manifest unknown
2026-05-18T13:49:43.3838247Z ##[error]Bash exited with code '1'.
2026-05-18T13:49:43.3838247Z ##[section]Finishing: Pre-pull AWF and MCPG container images (v0.25.44)

Timeline issue record:

{
  "type": "error",
  "category": "General",
  "message": "Bash exited with code '1'.",
  "data": { "source": "TaskInternal", "logFileLineNumber": "141" }
}

ghcr.io registry probe (anonymous pull token):

GET https://ghcr.io/v2/github/gh-aw-mcpg/manifests/v0.3.7  →  404 Not Found
GET https://ghcr.io/v2/github/gh-aw-mcpg/manifests/0.3.7   →  404 Not Found
Highest tag in /tags/list:  v0.2.16

Classification

• Category: bug — the ado-aw compiler emits an incorrect generated YAML that references a non-existent container image tag.
• Not a user-configuration issue: the failing step and the version pin are entirely produced by the compiler, with no user-tunable input.
• Not transient: the same step has failed identically on every run since at least 16 May, and the same v0.3.7 pin is still present on main and on the most recent release tag v0.30.2.

Recommended next steps (for maintainers — not implemented as part of this report)

• Either publish ghcr.io/github/gh-aw-mcpg:v0.3.7, or downgrade the MCPG tag baked into the compiler default to a tag that exists (e.g. v0.2.16).
• Cut a new ado-aw release so downstream lock files can be re-compiled.
• Consider adding a release-time check that all referenced container image tags resolve before publishing.