feat(compile): runtime prompt rendering via composable bash + awk (supersedes #617)

[jamesadevine]

Summary

Supersedes #617. Same goal — body-only edits to the agent .md should not require recompiling the pipeline — but a materially different mechanism. After inspecting how gh-aw renders prompts in its generated lock yamls (.github/workflows/*.lock.yml), it was clear that their approach is significantly more transparent and avoids the heavyweight Node bundle. This PR rebuilds the runtime-rendering path along the gh-aw lines.

What the compiled Agent job now emits (default, inlined-imports: false):

- bash: |
    set -euo pipefail
    OUT="/tmp/awf-tools/agent-prompt.md"
    mkdir -p "$(dirname "$OUT")"

    # 1. Compose: body from the workspace + extension supplements.
    {
      cat "$(Build.SourcesDirectory)/agents/foo.md"
      printf '\n\n'
      cat << '__ADO_AW_SUPP_SAFEOUTPUTS_EOF__'
      ---
      ## Important: Safe Outputs
      ...
      __ADO_AW_SUPP_SAFEOUTPUTS_EOF__
    } > "$OUT.raw"

    # 2. Strip the agent's YAML front matter.
    awk 'BEGIN { skip = 0 }
         NR == 1 && /^---$/ { skip = 1; next }
         skip && /^---$/    { skip = 0; next }
         !skip              { print }' "$OUT.raw" > "$OUT.body"

    # 3. Single-pass substitution: ${{ parameters.NAME }}, $(VAR),
    # \$(...) escape, $[...] warning. Values come from ENVIRON.
    awk '...single-pass walker...' "$OUT.body" > "$OUT"

    # 4. Fail closed on empty output.
    [ ! -s "$OUT" ] && { echo "..."; exit 1; }

    rm -f "$OUT.raw" "$OUT.body"
    echo "Agent prompt:" ; cat "$OUT"
  displayName: "Render agent prompt"
  env:
    ADO_AW_PARAM_TARGET: ${{ parameters.target }}

Everything is visible in the lock yaml: which body file is loaded, which supplements are appended (and their contents), how front matter is stripped, and how substitution works.

Why v3 supersedes v2 (#617)

PR #617 shipped a prompt.js ncc bundle driven by a base64-encoded PromptSpec. It worked, but:

	v2 (#617)	v3 (this PR)
Lock yaml readability	Opaque node /tmp/ado-aw-scripts/prompt.js + 4KB base64 spec	Plain bash + awk, all heredocs visible
Agent VM dependencies	NodeTool@0 install + scripts.zip download required for every default-mode pipeline	Zero new deps (bash + awk present on every ADO image)
Adding a new extension supplement	Update prompt_supplement() → goes into PromptSpec.supplements JSON array	Update prompt_supplement() → embeds as a labelled heredoc directly in YAML
Net diff	+1914 / -197 LOC	+1213 / -165 LOC
Files added	5 new TS files + IR + schema gen + smoke tests	2 new fixtures

The gh-aw lock yamls in this repo (e.g. .github/workflows/rust-pr-reviewer.lock.yml lines 160-220) use exactly this composable-bash shape. After looking at one carefully, v2's "neat symmetric IR" turned out to be solving a problem nobody had.

Single-pass substitution (security property preserved)

The awk substitution program walks the assembled prompt once, matching at each position any of:

Token	Resolved via	Notes
\$(VAR)	escape	Backslash stripped; $(VAR) literal.
${{ parameters.NAME }}	ENVIRON["ADO_AW_PARAM_<UPPER>"]	Only declared parameters substitute.
$(VAR) / $(VAR.SUB)	ENVIRON["<UPPER>"] (dot→underscore)	Unset vars left verbatim with a warning.
$[ ... ]	not substituted	Verbatim with one-shot warning.

The walker uses substring slicing rather than gsub() so replacement text is never re-scanned. This blocks the same chaining attack flagged in #395's bot review: if a caller queues with target = "$(System.AccessToken)", the substituted value lands in the rendered prompt as the literal string $(System.AccessToken), not the access token. Unit and integration tests assert this property directly.

Scope

• ✅ Body cat'd from workspace at runtime (the runtime-injection point).
• ✅ Extension supplements embedded as labelled heredocs in the same step (visible in the lock yaml).
• ✅ Front-matter strip via awk.
• ✅ Single-pass awk substitution (blocks the chaining attack).
• ✅ inlined-imports: true escape hatch (gh-aw-compatible field name).
• ✅ Flattened ado-script.zip layout: top-level gate.js, /tmp/ado-aw-scripts/gate.js.
• ✅ Shared needs_scripts_bundle() trait method on CompilerExtension (gate consumes; future bundles join the same dedupe).
• ✅ strip_prefix(" ") fix in the inlined heredoc branch (preserves author-supplied leading whitespace).
• ❌ No prompt.js bundle. No PromptSpec IR. No ADO_AW_PROMPT_SPEC env. No ExportPromptSchema CLI. No types-prompt.gen.ts.

Files

Rust

• src/compile/common.rs — collect_prompt_supplements, generate_prepare_agent_prompt (both branches), PromptSupplement struct (local, not in an IR file), supplement_delimiter. Templates' {{ agent_content }} → {{ prepare_agent_prompt }}.
• src/compile/extensions/mod.rs — needs_scripts_bundle() trait method, node_tool_step/scripts_download_step/scripts_install_steps_if_needed shared helpers.
• src/compile/extensions/trigger_filters.rs — refactored to declare needs_scripts_bundle().
• src/compile/filter_ir.rs — gate path constant updated.
• src/compile/types.rs — inlined-imports: bool field on FrontMatter.
• src/data/{base,1es-base,job-base,stage-base}.yml — {{ agent_content }} → {{ prepare_agent_prompt }}.

Tests

• tests/fixtures/runtime-prompt-default-agent.md (new)
• tests/fixtures/runtime-prompt-inlined-agent.md (new)
• tests/compiler_tests.rs — 4 new integration tests asserting body absence, awk substitution presence, supplement heredoc structure, and inlined-mode heredoc.
• src/compile/common.rs (test module) — 8 unit tests for generate_prepare_agent_prompt covering both branches, parameter env mappings, supplement embedding, the \$(...) escape, and the chaining-attack regression.
• src/compile/types.rs (test module) — 4 round-trip tests for the inlined-imports field.

CI / release

• .github/workflows/release.yml — flatten zip layout (top-level gate.js).
• .github/workflows/ado-script.yml — unchanged from origin/main (no types-prompt.gen.ts drift check needed).

Docs

• docs/ado-script.md — reworded intro to clarify prompt rendering is NOT a bundle; pointer to template-markers.md and front-matter.md.
• docs/front-matter.md — inlined-imports field documented; explanation describes the bash+awk shape.
• docs/template-markers.md — {{ agent_content }} → {{ prepare_agent_prompt }} with full description of the compose + strip + substitute pipeline.
• docs/extending.md — needs_scripts_bundle() trait method; supplement delivery via labelled heredocs.
• AGENTS.md — source-tree overview updated (no prompt_ir.rs, no prompt.js).

Test plan

• cargo build ✓
• cargo test — 1602 lib + 96 compiler_tests + sundry, 0 failures ✓
• cargo clippy --all-targets --all-features — no new lints (baseline pre-existed) ✓
• End-to-end spot-check on a copy of tests/fixtures/minimal-agent.md:
  • Default mode. YAML emits cat "$(Build.SourcesDirectory)/m.md", includes the SafeOutputs supplement heredoc, awk strip program, and awk substitution program. Body line is absent from YAML. No NodeTool@0, no scripts.zip download for prompt rendering, no JS bundle.
  • inlined-imports: true. YAML contains the body verbatim inside an AGENT_PROMPT_EOF heredoc; supplements emitted as separate cat >> steps via the unchanged wrap_prompt_append.

The awk substitution program is structurally identical to a battle-tested pattern (single-pass walker with substring slicing). I do not have a Linux awk locally to do an end-to-end runtime verification, but cargo test covers the YAML emission shape and the unit tests cover the single-pass property by construction (replacement text is fed through ENVIRON, never re-injected into the regex match).

Migration

Default behaviour changes. Existing compiled .lock.yml files will fail ado-aw check after this lands until consumers recompile.

inlined-imports: true is the documented escape hatch for users who can't recompile immediately, need a fully self-contained YAML, or are compiling outside the trigger repo.

[jamesadevine]

Closing per author request — the v3 design dropped the {{#runtime-import}} token mechanism (the actual runtime-injection script) that gh-aw uses, and the right thing is to restart rather than patch v3.

For future reference, what v3 does:

• Body IS read from the workspace at runtime via cat "$(Build.SourcesDirectory)/<path>.md" in the compose step — so prose-only edits to the source .md do not require recompiling. That part of the goal is met.
• Extension supplements are embedded as labelled heredocs in the same compose step.
• Front matter is stripped via awk and parameters/variables are substituted via a single-pass awk program.

What v3 does NOT do (and what motivated this PR being closed):

• No {{#runtime-import path}} token mechanism — gh-aw uses this to compose prompts from multiple workspace files (the body itself is referenced via {{#runtime-import .github/workflows/<agent>.md}}, and a interpolate_prompt.cjs step resolves it). v3 hardcoded a cat for the body and provided no story for sharing prose fragments across agents or for transitive imports.
• No interpolate step / "runtime injection script" — v3 inlined the body load into the compose step rather than making it a generic token expansion pass.

Future work should reintroduce the token mechanism (likely as an awk/sed pass or a small dedicated bundle, depending on how complex the resolution semantics need to be — nested imports, cycle detection, etc.).

[github-actions]

🔍 Rust PR Review

Summary: Looks good overall — the bash+awk design is sound and the security property (single-pass ENVIRON substitution) is well-preserved. A few small issues worth fixing before merge.

Findings

🐛 Bugs / Logic Issues

• src/compile/types.rs:698 — Stale docstring on inlined_imports. Line 698 reads "lets prompt.js read it from the workspace at runtime", but this PR removes prompt.js from the default path entirely — it's pure bash+awk now. The docstring should say something like "a bash+awk compose step reads it from the workspace at runtime". This will mislead contributors reading the field definition.

• tests/fixtures/runtime-prompt-default-agent.md:9 — The fixture body says "Instead, prompt.js reads it from the workspace at pipeline runtime...". The fixture body is expected not to appear in compiled output (that's what the test asserts), but the stale prompt.js reference in the fixture source is confusing when someone reads it.

• tests/compiler_tests.rs (test_prepare_agent_prompt_runtime_emits_param_env_mappings) — The test uses "target-branch" as a parameter name (with a hyphen). validate::is_valid_parameter_name rejects hyphens ([A-Za-z_][A-Za-z0-9_]*), so generate_parameters would bail before generate_prepare_agent_prompt is ever reached with a hyphenated name. The test exercises a code path that's unreachable in production. Should use targetBranch (or any valid ADO identifier) to match what the validator allows.

⚠️ Suggestions

• src/compile/extensions/mod.rs:808 (scripts_download_step) — Uses set -eo pipefail instead of set -euo pipefail. Every other generated bash step in the compiler uses -euo. Missing -u means unset variables silently expand to empty strings during the curl/unzip step; e.g. if dir were ever empty, mkdir -p and unzip -d would silently target . instead. Should be set -euo pipefail for consistency and defence-in-depth.

• src/compile/extensions/mod.rs:829-831 (bundle_path) — Marked #[allow(dead_code)] with a comment "Kept as a reusable helper for future bundle consumers". Suppressing a lint for speculative future use is a maintenance smell; it will remain stale indefinitely. Remove it now and re-add when a concrete consumer lands.

• src/compile/common.rs:3533 — generate_prepare_agent_prompt is called with &front_matter.parameters (user-declared only), not &parameters (which includes the auto-injected clearMemory). This means ${{ parameters.clearMemory }} in a prompt body won't substitute at runtime — it will be left verbatim with a warning. This is probably intentional, but is undocumented. A one-line comment at the call site would prevent future confusion.

✅ What Looks Good

• Single-pass awk via ENVIRON is exactly the right primitive — replacement values are never re-injected into the regex walk, which closes the chaining attack cleanly without any special-case logic.
• supplement_delimiter validation via anyhow::ensure! prevents heredoc delimiter collision at compile time; the error message is actionable.
• replace_with_indent interaction with YAML block-scalar stripping is handled correctly: all bash-body lines carry the same 4-space base indent so the post-YAML-strip heredoc close delimiter lands at column 0.
• Test depth is good: 8 focused unit tests + 4 integration tests cover both branches, the chaining-attack regression, and the strip_prefix fix.
• The refactor of TriggerFiltersExtension to declare needs_scripts_bundle() instead of emitting its own NodeTool + download steps is a clean separation of concerns.

Generated by Rust PR Reviewer for issue #623 · ● 1.6M · ◷