feat(compile): runtime prompt injection via prompt.js bundle

[jamesadevine]

Summary

Replaces compile-time prompt embedding with runtime prompt injection via a new prompt.js ado-script bundle. The default behaviour changes: the agent's markdown body is no longer baked into the compiled pipeline YAML — it's read from the workspace at runtime by prompt.js, which also strips front matter, appends extension supplements, and substitutes pipeline parameters / variables.

UX win: body-only edits to the agent .md no longer require recompiling the pipeline. Front-matter changes still recompile (correct: front-matter is configuration, body is prompt).

Supersedes #395. That PR was 155 commits behind main and predated the gate.js/ado-script.zip infrastructure from #389. This is a fresh rebuild on current main that:

• Folds in the post-feat(compile): replace Python gate evaluator with bundled TypeScript gate.js #389 zip + checksum-verified install pattern.
• Flattens the release-zip layout so bundles live at top-level (/tmp/ado-aw-scripts/gate.js, /tmp/ado-aw-scripts/prompt.js) instead of leaking the internal ado-script/dist/<bundle>/index.js path. Versioned zip → no cross-version contract → clean cutover.
• Hoists the NodeTool@0 + scripts-download install pair into a shared compiler-level emission so future bundle consumers don't each carry their own download (needs_scripts_bundle() on the CompilerExtension trait).
• Addresses the bot-review findings on feat(compile): runtime prompt injection via prompt.js bundle #395:
  • Single-pass substitution in prompt.js (blocks the parameter-value-into-$(VAR) chaining attack).
  • strip_prefix(" ") instead of trim_start_matches(' ') in the inlined branch (preserves author-supplied leading whitespace on the first body line).
  • Integration fixture in tests/compiler_tests.rs exercising the full compile_shared round-trip with body-absent and PromptSpec-present assertions.

Set inlined-imports: true in front matter to opt out and keep the legacy heredoc-embedded behaviour. Use this for self-contained YAML, restricted networks, or debugging.

Design

The runtime contract is a new PromptSpec IR mirroring the proven GateSpec pattern:

• src/compile/prompt_ir.rs — PromptSpec, PromptSupplement, PROMPT_SPEC_VERSION = 1, schemars-derived JSON Schema.
• Hidden CLI subcommand export-prompt-schema (mirrors export-gate-schema).
• npm run codegen now produces both types.gen.ts (gate) and types-prompt.gen.ts (prompt). CI drift check covers both.
• PromptSpec is JSON-serialized + base64-encoded into a single ADO_AW_PROMPT_SPEC env var on the prompt.js step. Each declared parameter gets a separate ADO_AW_PARAM_<NAME>: ${{ parameters.<NAME> }} env entry — values stay outside the spec so ADO secret-redaction and late binding work correctly.

Substitution semantics (single-pass)

prompt.js walks the rendered prompt body+supplements exactly once with one regex matching all supported token shapes. Replacement values are returned verbatim and never re-scanned:

Pattern	Resolved via	Notes
\$(VAR) / \$(VAR.SUB)	escape	Backslash stripped; $(...) left literal.
${{ parameters.NAME }}	env ADO_AW_PARAM_<NAME upper, hyphen→underscore>	Only declared parameters substitute; others left verbatim with a warning.
$(VAR) / $(VAR.SUB)	env <name upper, dot→underscore> (ADO native)	Unset variables left verbatim with a warning. Secrets aren't auto-exposed and stay verbatim.
$[ ... ]	not substituted	Left verbatim with one warning per render.

Single-pass is load-bearing. It blocks the "queue-with-malicious-parameter-value" chaining attack: if a caller queues with target = "$(System.AccessToken)", the substituted value lands in the rendered prompt as the literal string $(System.AccessToken), not the access token itself. Unit and smoke tests assert this property directly.

Pipeline ordering

When inlined-imports: false (default), the new {{ prepare_agent_prompt }} template marker emits a three-step bundle in the Agent job:

1. NodeTool@0 to install Node 20.x (timeout-capped at 5 minutes).
2. bash: download of ado-script.zip (with checksums.txt SHA-256 verify) and unzip into /tmp/ado-aw-scripts/.
3. bash: node /tmp/ado-aw-scripts/prompt.js with the spec env + per-parameter env mappings.

When inlined-imports: true, the same marker emits the legacy heredoc step embedding the body verbatim, plus per-extension cat >> supplement steps via the existing wrap_prompt_append.

Scope

• ✅ Body injection from workspace .md.
• ✅ Extension prompt supplements consolidated in PromptSpec.supplements.
• ✅ Variable substitution for ${{ parameters.* }} and $(VAR) patterns, single-pass.
• ✅ Shared scripts-bundle install via needs_scripts_bundle() on the extension trait, dedup'd within each consuming job.
• ✅ Flattened zip layout: top-level gate.js / prompt.js.
• ❌ {{#import path }} runtime-import macros — out of scope; the spec is additive-friendly so this is not a one-way door.
• ❌ Threat-analysis prompt stays compile-time embedded — it's compiler-internal, not user-visible.

Files changed

Rust (compile-time)

• src/compile/prompt_ir.rs (new) — PromptSpec, PromptSupplement, PROMPT_SPEC_VERSION, schema generator.
• src/compile/mod.rs — pub(crate) mod prompt_ir;.
• src/compile/types.rs — adds inlined_imports: bool (#[serde(rename = "inlined-imports", default)]) to FrontMatter + 4 round-trip tests.
• src/compile/common.rs — new helpers collect_prompt_supplements, generate_prepare_agent_prompt; generate_prepare_steps takes inlined_imports: bool and skips wrap_prompt_append in the runtime branch; compile_shared builds and emits {{ prepare_agent_prompt }}. Includes the strip_prefix(" ") fix.
• src/compile/extensions/mod.rs — new needs_scripts_bundle() trait method, node_tool_step + scripts_download_step + scripts_install_steps_if_needed shared helpers.
• src/compile/extensions/trigger_filters.rs — refactored to declare needs_scripts_bundle() instead of inlining the install pair.
• src/compile/filter_ir.rs — gate path constants updated to flattened /tmp/ado-aw-scripts/gate.js.
• src/main.rs — adds hidden Commands::ExportPromptSchema subcommand.
• src/data/{base,1es-base,job-base,stage-base}.yml — replace static heredoc step with {{ prepare_agent_prompt }}.

TypeScript (runtime bundle)

• scripts/ado-script/src/prompt/index.ts (new) — entry point; decodes ADO_AW_PROMPT_SPEC, validates version, reads source, strips front matter, assembles body+supplements, runs single-pass substitute, writes output.
• scripts/ado-script/src/prompt/frontmatter.ts (new) — pure stripFrontMatter mirroring parse_markdown_detailed semantics from Rust.
• scripts/ado-script/src/prompt/substitute.ts (new) — single-pass substitution engine (regex with named groups; values never re-scanned).
• scripts/ado-script/src/prompt/__tests__/{frontmatter,substitute}.test.ts (new) — 21 unit tests including the parameter-chains-into-VAR attack case.
• scripts/ado-script/src/shared/types-prompt.gen.ts (new, AUTO-GENERATED) — committed; CI verifies it stays in lockstep with the Rust IR.
• scripts/ado-script/test/prompt-smoke.test.ts (new) — 4 end-to-end smoke tests against the ncc-built bundle including the chained-substitution attack.
• scripts/ado-script/package.json — extends codegen (now codegen:gate + codegen:prompt); adds build:prompt; updates build and build:check to cover both bundles.
• scripts/ado-script/src/gate/predicates.ts — error-message reference updated for the flattened bundle path.

CI / infra

• .github/workflows/ado-script.yml — drift check now covers both types.gen.ts and types-prompt.gen.ts; build step builds both bundles; smoke step runs both bundles.
• .github/workflows/release.yml — package step flattens dist/<bundle>/index.js → top-level <bundle>.js inside ado-script.zip.

Docs

• docs/ado-script.md — new "What prompt.js does" section, updated workspace layout, updated codegen section, updated download-wiring section.
• docs/front-matter.md — documents inlined-imports: true and the rendering model.
• docs/template-markers.md — replaces {{ agent_content }} with {{ prepare_agent_prompt }} and documents the substitution contract.
• docs/extending.md — adds needs_scripts_bundle() trait method and notes that prompt_supplement() content travels through PromptSpec.supplements (not heredoc steps) by default.
• docs/filter-ir.md — gate paths updated to the flattened layout.
• AGENTS.md — adds prompt.js and prompt_ir.rs to the source-tree overview.

Test fixtures

• tests/fixtures/runtime-prompt-default-agent.md (new) — drives the runtime-path integration assertions.
• tests/fixtures/runtime-prompt-inlined-agent.md (new) — drives the inlined-path integration assertions.
• tests/compiler_tests.rs — 4 new integration tests covering both modes + PromptSpec spec decoding + body-absence.

Test plan

• cargo build ✓
• cargo test ✓ — 1605 lib + 95 compiler_tests + smaller test files, 0 failures. Includes:
  • 5 new prompt_ir tests (schema/version/serialization).
  • 4 new inlined-imports field tests in compile::types.
  • 8 new generate_prepare_agent_prompt / collect_prompt_supplements tests in compile::common.
  • 7 refactored trigger_filters tests covering the dedupe.
  • 4 new integration tests in compiler_tests.rs (asserts on PromptSpec base64 contents, body absence, heredoc presence).
• cargo clippy --all-targets --all-features ✓ — no new lints (baseline pre-existed).
• cd scripts/ado-script && npm test ✓ — 214 tests pass (193 baseline + 21 new for prompt.js: 8 frontmatter + 13 substitute).
• cd scripts/ado-script && npm run typecheck ✓ — clean.
• cd scripts/ado-script && npx vitest run -c vitest.config.smoke.ts ✓ — 6 tests pass (2 gate.js smoke + 4 new prompt.js end-to-end including the chained-substitution attack).
• Manual end-to-end spot-check on examples/sample-agent.md:
  • Default branch: compiled YAML emits node /tmp/ado-aw-scripts/prompt.js + ADO_AW_PROMPT_SPEC env, the prompt body is absent from the YAML, and decoding the base64 spec confirms source_path resolves to $(Build.SourcesDirectory)/sample-agent.md ({{ trigger_repo_directory }} substituted before encoding).
  • Inlined branch: with inlined-imports: true, YAML contains the body verbatim inside an AGENT_PROMPT_EOF heredoc and does not invoke prompt.js.

Migration

Default behaviour changes. Existing compiled .lock.yml files will fail ado-aw check after this lands until the consumer recompiles. The integrity check property strengthens: with the body no longer embedded, prose-only edits no longer drift from the compiled YAML at all.

inlined-imports: true is the one-line escape hatch for users who can't recompile immediately, can't reach github.com from the Agent pool, or want a fully self-contained pipeline file.

[github-actions]

🔍 Rust PR Review

Summary: Good architecture with one load-bearing runtime bug that will cause the default path to always fail in a real ADO pipeline.

---

Findings

🐛 Bugs / Logic Issues

source_path is an ADO expression, not a filesystem path — prompt.js can't open it

scripts/ado-script/src/prompt/index.ts lines 57–63; src/compile/common.rs generate_prepare_agent_prompt

generate_trigger_repo_directory returns the string "$(Build.SourcesDirectory)" (or "$(Build.SourcesDirectory)/$(Build.Repository.Name)"). After the compile-time replace this value is embedded verbatim in the JSON spec and then base64-encoded. Because the base64 payload contains no literal $(...) text visible to ADO's YAML pre-processor, ADO never expands it. prompt.js decodes the spec and immediately does:

if (!fs.existsSync(spec.source_path)) { ... process.exit(1); }
const source = fs.readFileSync(spec.source_path, "utf8");

At that point spec.source_path is still the literal string "$(Build.SourcesDirectory)/agent.md". That is not a valid filesystem path; the step exits with "Source markdown not found" on every real ADO run.

The fix is to resolve ADO variable references in source_path before using it as a path. Because substitute.ts already knows how to map $(Build.SourcesDirectory) → process.env.BUILD_SOURCESDIRECTORY (and $(Build.Repository.Name) → BUILD_REPOSITORY_NAME etc.), the minimal fix is a targeted resolve in index.ts:

// Resolve ADO variable refs in the path before touching the filesystem.
const resolvedSourcePath = substitute(spec.source_path, [], (msg) => logWarning(msg));
if (!fs.existsSync(resolvedSourcePath)) { ... }
const source = fs.readFileSync(resolvedSourcePath, "utf8");

The smoke tests don't catch this because they set source_path to a real temp-dir path, not to "$(Build.SourcesDirectory)/...".

---

⚠️ Suggestions

needs_scripts_bundle() / setup_steps() logic is duplicated in TriggerFiltersExtension

src/compile/extensions/trigger_filters.rs

needs_scripts_bundle() independently recomputes whether any filter steps exist (via lower_pr_filters / lower_pipeline_filters) to decide whether to claim the shared bundle install. setup_steps() recomputes the same thing independently. If they ever diverge:

• needs_scripts_bundle() → true but no gate steps → wasted NodeTool + download in the Setup job.
• needs_scripts_bundle() → false but gate steps present → gate.js is invoked without the bundle being downloaded, causing a job failure.

Consider extracting a single has_gate_steps(&self) -> bool method used by both, or simply have needs_scripts_bundle call !self.setup_steps_internal().is_empty() if the duplication grows.

scripts_download_step uses set -eo pipefail while the prompt.js invocation step uses set -euo pipefail

src/compile/extensions/mod.rs scripts_download_step()

The download step has no shell variables so the missing -u has no practical impact today, but it's inconsistent and worth aligning for future edits to the download script.

---

✅ What Looks Good

• Single-pass substitution is correctly load-bearing: the regex replaces in one pass and String.prototype.replace with a function callback never re-scans the returned value — the "queue with $(System.AccessToken) as a parameter value" attack is genuinely blocked, and smoke tests assert it.
• strip_prefix(" ") over trim_start_matches(' ') fix is semantically correct and well-commented.
• Compile-time source_path validation (must start with $(Build.SourcesDirectory)) gives an early, clear error for agents compiled outside their trigger repo; the inlined-imports: true escape hatch is well-motivated.
• Base64-encoding the entire spec correctly sidesteps template injection — ${{ }} and $(...) in supplement content can't escape the base64 envelope at the YAML level.
• scripts_install_steps_if_needed dedup (one NodeTool@0 + download per job) is clean, and the comment explaining why the Agent job must re-download (different VM) is accurate and appreciated.
• Test coverage is solid: the inlined-imports round-trip tests, generate_prepare_agent_prompt unit tests, and the four integration tests with base64 spec decoding are all high-value additions.

Generated by Rust PR Reviewer for issue #617 · ● 795.1K · ◷

[jamesadevine]

Superseded by #623, which rebuilds this work along gh-aw lines after observing how their generated lock yamls compose prompts (.github/workflows/*.lock.yml in this repo). #623:

• Drops the prompt.js ncc bundle, the PromptSpec IR, the ADO_AW_PROMPT_SPEC env contract, the export-prompt-schema CLI subcommand, and the types-prompt.gen.ts codegen.
• Replaces them with a single inline bash + awk step in the compiled YAML: cat "$(Build.SourcesDirectory)/<path>/agent.md" for the body, embedded heredocs for each extension supplement (visible directly in the lock yaml), awk front-matter strip, and a single-pass awk substitution program.
• Preserves the chained-substitution attack defence: the awk walker uses substring slicing rather than gsub(), so replacement text is never re-scanned.

Net diff: 1213 / 165 in #623 vs 1914 / 197 here. No Agent-VM Node install for prompt rendering. Net win on lock-yaml readability.

Closing in favour of #623.