How to prove the template repo is healthy, how to prove a generated project actually deploys
to Vertex AI Agent Engine, and how to move an existing agent into it. This is a runbook:
re-run the relevant section whenever {{cookiecutter.project_slug}}/ changes.
- Read
engine.api_resource.spec.service_account, neverengine.service_account. The latter returnsNoneregardless of the real value — trusting it produces a false claim that an agent has no service account set when it actually does. - Check a log sink's
disabledflag, not just its filter.gcloud logging sinks describe _Default --format='value(disabled)'. A disabled_Defaultdiscards every non-audit entry however written, including a directgcloud logging write, and looks exactly like broken instrumentation. Notegcloud logging exclusionsis not a valid subcommand in this CLI version, so an exclusions check can error in a way that reads as "none found". - Sink changes take several minutes to propagate, and entries written meanwhile are lost. This produces false negatives. Write a canary and poll for it before concluding anything.
gsutil -m rmhangs on macOS (its own output warns: bugs.python.org/issue33725). The deletes complete but the process never exits. Usegcloud storage rm -r, or drop-m. Do not conclude the deletes failed — verify withlsafter interrupting.- Tool spans are children of ADK's
invoke_workflowroot span, so aview=ROOTSPANtrace listing never shows them.read_traces.py --spansexpands them;--filter "span:<tool>"selects traces containing one. - The requirements warning is non-deterministic — one deploy can report a different
missing-package count than another for identical code. Only
cloudpickleis auto-appended; the rest are satisfied transitively. - ADC is a separate credential store from the CLI login.
gcloud auth logindoes not refresh Application Default Credentials —vertexai.init()reads ADC, so a local deploy also needsgcloud auth application-default login; skipping it surfaces mid-deploy as aRefreshError. Both stores go stale independently, so check both before deploying:gcloud auth print-access-tokenandgcloud auth application-default print-access-token.
DeploymentConfig derives the bucket and SA from the project id, so a deploy needs one line of
.env:
echo GOOGLE_CLOUD_PROJECT=example-gcp-project > .env
# add AGENT_ENGINE_RESOURCE_NAME=<resource> to UPDATE rather than create a second billable one
make deploy-devTeardown, in this order — the engine first, since it is the billable part:
uv run python -c "
import vertexai; from vertexai import agent_engines
vertexai.init(project='example-gcp-project', location='europe-west1')
a = agent_engines.get('<resource>')
assert a.display_name == '<expected>' # guard against deleting a sibling
agent_engines.delete('<resource>', force=True)"
gcloud storage rm -r "gs://example-gcp-project-agent-staging/<agent-slug>/" # NEVER the bucket root
gcloud monitoring dashboards delete <id> --project=example-gcp-project --quiet
gcloud monitoring policies delete "projects/example-gcp-project/alertPolicies/<id>" --project=example-gcp-project --quietgcloud monitoring policies delete takes one policy per invocation — passing two prints
help and silently does nothing.
- An AI coding agent working in this repo is typically denied
rm,curl,git commit,git push, andRead/Writeon.env*(including.env.example), by design. Practical effects: teardown deletions, version lookups,.envedits, and commits are always a manual step for a human. The.env*denial also silently strips matches from a recursivegreprun through such an agent's shell tool, with no error — never conclude "nothing references X" from a grep that could have hit.env*. - Validate commit messages before committing:
uv run cz check --commit-msg-file <file>. The commit-msg hook runs commitizen. Accepted types are the 12 fromcz_conventional_commits. uv --directorysets the working directory, so a generated project lands wherever that points rather than where you ran the command. Use--output-dirwith an absolute path, orcdinto the target first and checkpwd.gcloud alphais not installed in a typical environment, so monitoring channels/policies must be queried through the REST API. See §3 for workingAuthorizedSessionsnippets.- Generate with
cookiecutter .when testing uncommitted template changes.cruft create <github-url>fetches the pushed branch, so it would silently test the old code — the opposite of what you want, unless the test specifically involves GitHub CI (see below). - A stale
cdafter regenerating a scratch project is a real hazard. Moving the old project aside leaves the terminal inside the moved directory, somake deploy-devwould deploy pre-change code. Re-cdto the absolute path and checkpwdfirst.
GitHub disables workflows on forks until someone clicks "I understand my workflows, go ahead
and enable them" in the Actions tab, and there is no REST endpoint for that button. The API
is actively misleading: actions/permissions reports "enabled": true and every workflow
shows active, while actions/runs returns total_count: 0 — zero runs, ever.
A freshly created repo is not a fork, so its workflows run on the first push. That sidesteps the button entirely. Two scratch repos cover the two distinct workflow sets, which barely overlap:
| Workflow | Template repo | Generated project |
|---|---|---|
ci.yml |
✓ lints hooks/ |
✓ 4 jobs, different file |
lint-pr.yml |
✓ | ✓ |
validate-template.yml |
✓ | ✗ does not exist |
eval.yml, cruft-check.yml, deploy.yml, security.yml |
✗ | ✓ |
So a generated dummy can never exercise validate-template.yml. To test it, push a copy of
the template repo to a fresh non-fork repo.
# template-repo CI
git clone /path/to/agent-deployment-template /tmp/ci-template
cd /tmp/ci-template && git remote remove origin
gh repo create adk-template-ci-check --private --source=. --push
# generated-project CI — use cruft, not cookiecutter (see below)
cd /tmp && uv --directory /path/to/agent-deployment-template run \
cruft create https://github.com/philippschaedle/agent-deployment-template \
--no-input -y --extra-context '{"project_name":"Ci Check Agent"}'uv --directory sets the working directory, so the generated project lands in the
template repo, not where you ran the command — cd into the target and use an absolute
--directory, or just move the result afterwards.
Generate with cruft create <url>, not cookiecutter ., whenever a test involves
GitHub. Plain cookiecutter writes "template": "." and an empty commit, which CI cannot
resolve, and it hides two real bugs that only the cruft path exhibits (the detect-secrets
.cruft.json finding and a dirty tree).
lint-pr.yml, eval.yml and cruft-check.yml are pull_request-only — a push to main
will not fire them. Open a PR in the scratch repo to exercise them, and note gh pr checks
needs the PR number when --repo is given: gh pr checks 1 --repo <owner/repo> --watch.
Run these in order. Each one gates the next — don't skip ahead if one fails.
make install # uv sync (dev deps for hooks/) + installs the pre-commit and
# commit-msg git hooks — easy to forget in a fresh clone,
# and nothing runs on commit until they are installed
make lint # ruff check hooks/
make format # ruff format hooks/ (then re-run lint if it changed anything)
make typecheck # pyright hooks/
make pre-commit # every pre-commit hook, all filesNote the template repo's hook set is deliberately leaner than the generated project's:
ruff, ruff-format, check-yaml, check-json, end-of-file-fixer, trailing-whitespace,
check-merge-conflict, markdownlint-cli2, plus commitizen at the commit-msg stage. The
generated project additionally runs pyright, detect-secrets, check-toml and
check-added-large-files (see {{cookiecutter.project_slug}}/.pre-commit-config.yaml) —
don't expect those to fire here. pyright is covered by make typecheck instead.
make validateThis is the one command that matters most before a PR — it's exactly what ci.yml and
validate-template.yml run in CI. It generates a throwaway project into /tmp/cc-validate,
runs uv sync --frozen, ruff check ., ruff format --check ., pytest tests/unit, and
pytest tests/integration inside it, then deletes it. If this passes locally, CI will
almost certainly pass too — run it before pushing, not after.
Know what it still doesn't cover, because this is where bugs hide. make validate runs
neither pyright, nor tests/evals, nor the generated project's own pre-commit hook set,
and it only ever generates the default cookiecutter variant. Bugs have repeatedly lived in
exactly that blind spot. CI's validate-template.yml now covers the pre-commit hooks and four
variants; locally, 1c and 1d below are what close the gap.
Note also that ruff can never be run against the template sources in place —
{{cookiecutter.project_slug}}/pyproject.toml has unrendered Jinja in requires-python, so
ruff aborts with Failed to parse version: >={{cookiecutter.python_version}}. Anything that
lints or formats the generated tree must go through a rendered project. If you ever need to
format a template source by hand, use ruff format --isolated --line-length 88 --target-version py311 <file>, then confirm the result against a fresh generation — a
cookiecutter substitution changes a line's length, so template source and rendered source can
legitimately format differently. Keep length-sensitive Jinja off assertion lines (bind it to a
variable first, as tests/unit/test_config.py does).
Rather than relying only on make validate's temp-dir-and-delete cycle, generate one
persistent copy so you can poke at it in the following sections:
rm -rf /tmp/agent-test
cookiecutter . --no-input \
--output-dir /tmp/agent-test \
project_name="Test Agent" \
author_email="you@example.com" \
gcp_project_id="<a real or throwaway GCP project id>"
cd /tmp/agent-test/test-agent
make install
make lint
make typecheck # pyright, generated project — not covered by `make validate`
make test # unit + integration, with coverage
make eval # promptfoo — needs GOOGLE_API_KEY (or the provider set in .env) exported
make pre-commitmake eval needs an API key for whichever MODEL_PROVIDER you generated with (default
google → GOOGLE_API_KEY). If you don't want to spend a key on this pass, skip it here —
it's exercised again in Phase 2 below regardless, and CI's eval.yml gates PRs to
generated repos, not this one.
Why this matters: make pre-commit on a freshly generated project is what catches bugs
make validate alone won't — a freshly generated project failing its own hook set is a real,
recurring failure mode, not a hypothetical. Always run it against a fresh generation, not just
inside make validate's throwaway copy.
The make validate / manual-generate paths above use the defaults (model_provider=google,
python_version default, open_source_license default). Cookiecutter choice variables are
a combinatorial surface that a single default run doesn't cover. At minimum, repeat 1c's
make install && make lint && make typecheck && make test for:
-
model_provider=anthropicandmodel_provider=openai— both generate and pass cleanly, but note these do not exercise theLiteLlmbranch indeployment/config.py:resolve_model: that branch is selected at runtime from theMODEL_PROVIDERenvironment variable, and the cookiecutter answer never reaches it — so generating a project per provider cannot exercise those branches. They're covered bytests/unit/test_config.py, the only thing that can cover them. -
open_source_license=Proprietary— confirmLICENSEis removed byhooks/post_gen_project.pyand nothing else references it. Also runApache-2.0: the shared Jinja for the license banner is easy to break with a careless strip marker, and indentation only shows up wrong in the rendered output. - a non-default
python_version(e.g. 3.12) — this is the axis most likely to expose a ruff/pyright rule that differs by target version.
You don't need the full cross-product — one variant per changed cookiecutter axis is enough to catch a broken Jinja conditional.
The gate per variant should be stricter than the defaults above — ruff check +
ruff format --check + pyright + the full test suite + pre-commit run --all-files, plus a
check that the git tree is clean afterward — because a weaker gate is precisely what lets bugs
through. validate-template.yml runs four of these variants in CI on every PR, so this no
longer depends on someone remembering to do it by hand locally.
cd /tmp/agent-test/test-agent
cp .env.example .env
# fill in GOOGLE_API_KEY (or your chosen provider's key) in .env
make dev # opens http://localhost:8000, ADK web UI- Send a normal message, confirm a coherent reply comes back
- Trigger
get_current_datetimeandweb_searchexplicitly ("what time is it", "search for X") — confirm both tools actually fire and return results, not just that the model claims to have called them. Note: with noSERPAPI_API_KEYset,web_searchreturns a stubSearchResultunconditionally (seeagent/tools/example_tools.py) — the model faithfully reporting "I can't search right now" in that case is correct behavior, not a bug; don't mistake it for a tool failure. - Try one prompt-injection-style input and one PII-bearing input — confirm
prompts/system/safety.mdguidance actually holds in a live conversation, not just in the promptfoo dataset (tests/evals/datasets/skills/safety_*.jsonl) — the eval suite tests the prompt, this tests the running agent with real model sampling variance. - Watch stdout while doing this — confirm
@instrumentJSON log lines appear for each tool call (agent/observability.py) and look correctly shaped (see the log-field table in the generatedCLAUDE.md). This only applies to tool-invoking messages (datetime/search) — the injection/PII probes above don't call a tool, so there's nothing to check for PII leakage in@instrumentlog fields from those; that check only matters when PII flows through an instrumented tool call's arguments.
Cookiecutter-templated docs drift from the code they describe more easily than normal docs
(no compiler catches a stale Jinja block). Read the generated README.md and CLAUDE.md
top to bottom against the actual generated tree and confirm:
- Every make target the docs mention exists in the generated
Makefileand does what's described, and every target in theMakefileis documented in at least one of README/CLAUDE.md's tables. - The environment variable table matches what
deployment/config.pyand.env.exampleactually reference. -
AGENTS.md/CONTRIBUTING.md/SECURITY.mdrender without leftover{{ }}/{% %}template syntax anywhere — also checkREADME.md,CLAUDE.md,CHANGELOG.md,LICENSE,pyproject.toml.
This is the section most likely to hide bugs that only show up against real GitHub
infrastructure — a generated project that never actually committed anything, an untracked
uv.lock, split CI jobs held to the wrong coverage bar, an eval job that silently never ran.
Static inspection of the template can't catch any of these; only a real push/PR can.
${{ ... }} is Jinja before it is GitHub Actions. Any workflow added under
{{cookiecutter.project_slug}}/.github/workflows/ must wrap GitHub expressions in
{% raw %}...{% endraw %}, or generation fails outright with 'secrets' is undefined.
Trigger matrix. On a PR to main in a generated project, ci.yml (4 jobs), lint-pr.yml,
eval.yml and cruft-check.yml run — 7 checks. deploy.yml (push to main + manual
dispatch) and security.yml (push + weekly cron) do not run on PRs.
Full check names, for anyone configuring required status checks on a generated repo:
Lint, format & type-check
Unit tests
Integration tests (mocked LLM)
Coverage (full suite)
Check PR title format
Prompt Security Evaluation
Template drift check (non-blocking)
cruft-check.yml crashes with BadName: Ref '' did not resolve to an object when
.cruft.json's commit is empty — which is what plain cookiecutter . --no-input produces.
Left unhardened by decision: anyone following the documented cruft create gh:org/repo flow
gets a real SHA and never hits it.
This is the only way to know the Vertex AI Agent Engine path actually works; nothing above touches GCP.
This shapes every step below, and getting it wrong is how you break a sibling agent. Some resources are shared across every agent in the project; the rest are per-agent:
| Resource | Scope | Name |
|---|---|---|
| Service account | shared | agent-engine-sa@<project> |
| Staging bucket | shared | gs://<project>-agent-staging |
| Staging subfolder | per-agent | gs://<project>-agent-staging/<project_slug>/ |
| Agent Engine resource | per-agent | display name = project_name |
| Dashboard | per-agent | "<project_name> — Agent Engine" |
| Alert policies | per-agent | "<project_name> — high error rate" / "— high p95 latency" |
config.py states the intent directly — gcs_dir_name=_project_name(), commented "so
artifacts land at <bucket>/<project_slug>/ instead of the generic default". And
setup_gcp.sh's "already exists, skipping" is not a fallback: it is how the second and every
later agent deliberately adopts the shared SA and bucket.
So a shared team project is a valid target. If it already holds agent-engine-sa and a
staging bucket from an earlier agent, a new agent joins alongside rather than colliding. What
is not safe is deleting the shared pieces during teardown — see the Teardown section, which
is the one place this model genuinely changes what you run.
Two consequences worth knowing:
- One shared identity means no per-agent isolation. Every agent runs as
agent-engine-sawithaiplatform.user,logging.logWriterandcloudtrace.agentproject-wide, plusobjectAdminon the whole staging bucket. You cannot revoke or scope one agent independently, and Cloud Logging attributes every agent's activity to the same principal. Fine for a team dev project; wrong if two agents ever have different trust levels.SA_NAMEis hardcoded insetup_gcp.shwith no override, so per-agent service accounts are not currently possible without editing the script. make setup-monitoringmay attach alert policies to an existing notification channel. The display names are per-agent so they will not overwrite a sibling's, but if anyone is on-call for the project, check the channels first or skip that step.
The script ends with gcloud iam service-accounts keys create and echoes the base64 key to
stdout, where it lands in terminal scrollback. Standard GCP conventions say never to create
or download a service-account key file. That key exists only so deploy.yml can authenticate
in GitHub Actions; a local make deploy-dev uses ADC and never reads it.
Standing template finding, not yet fixed: the key flow should be replaced with Workload
Identity Federation. Until then, if you run setup-gcp, delete the key immediately —
server-side as well as on disk:
KEY_ID=$(python3 -c "import json;print(json.load(open('gcp-sa-key.json'))['private_key_id'])")
gcloud iam service-accounts keys delete "$KEY_ID" \
--iam-account="agent-engine-sa@$ADK_TEST_PROJECT.iam.gserviceaccount.com" \
--project="$ADK_TEST_PROJECT" --quiet
rm gcp-sa-key.jsonDeleting it server-side also makes the scrollback copy inert. Note it is a key on the shared service account, so a leak would affect every agent in the project.
Generate via cruft — the documented flow — so .cruft.json is realistic:
export ADK_TEST_PROJECT=example-gcp-project # or a fresh throwaway project
mkdir -p /tmp/agent-deploy && cd /tmp/agent-deploy
uv --directory /path/to/agent-deployment-template run \
cruft create https://github.com/philippschaedle/agent-deployment-template \
--no-input -y --extra-context "{\"project_name\":\"Deploy Test Agent\",\"gcp_project_id\":\"$ADK_TEST_PROJECT\"}"
# uv --directory sets the cwd, so the project lands in the template repo:
mv /path/to/agent-deployment-template/deploy-test-agent /tmp/agent-deploy/
cd /tmp/agent-deploy/deploy-test-agent
git status --short # expect empty — also confirms the cruft dirty-tree fix
cp .env.example .env
# GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION are pre-filled from the answers above.
# Add: GCS_STAGING_BUCKET=gs://<project>-agent-staging
# GOOGLE_API_KEY is NOT needed — the deployed agent runs on Vertex under the SA.Confirm the target before spending anything:
gcloud projects describe "$ADK_TEST_PROJECT" --format='value(projectId,lifecycleState)'
gcloud billing projects describe "$ADK_TEST_PROJECT" --format='value(billingEnabled)'make setup-gcp ENV=dev- Expect "already exists, skipping" for the SA and bucket when targeting a project that already hosts an agent. That is correct, not a warning.
- If you run
setup-gcp, delete the generated key immediately — server-side as well as on disk (commands above). - Confirm
setup-gcpis idempotent — run it twice against a project that already has these resources and confirm no duplicates or errors. Separately, verify the fresh-create path against a genuinely empty project — this is easy to skip if every project you test against already hosts an agent.
make deploy-dev- Watch it:
vertexai.init→ picklesroot_agent→ creates the Agent Engine resource → runs its own smoke test (deployment/scripts/health_check.py) → writes.agent_engine_resource - Confirm
AGENT_ENGINE_RESOURCE_NAME=...is printed at the end — that is whatdeploy.ymlcaptures - Confirm the artifacts landed under this agent's own subfolder,
gs://$ADK_TEST_PROJECT-agent-staging/deploy-test-agent/, and left any sibling's subfolder untouched
echo "AGENT_ENGINE_RESOURCE_NAME=$(cat .agent_engine_resource)" >> .env
make health-check # standalone, no fresh deploy; expect exit 0
make logs
make traces- Confirm
make logsactually surfaces the smoke test's@instrumentJSON lines and thatmake traces's printed filter matches real spans — not merely that the commands exit 0. Note a shared project makes this easier to misread: filter by this agent's resource name orjsonPayload.agent_name, or a sibling agent's traffic will look like yours.
If make logs / make traces come back empty against a resource you know just took traffic,
compare directly against the metrics resource type:
gcloud logging read 'resource.type="aiplatform.googleapis.com/ReasoningEngine"' \
--project="$ADK_TEST_PROJECT" --limit=5-
make setup-monitoringtwice — confirm the dashboard and both alert policies appear under the project, that their names carry this agent'sproject_name, and that a second run updates rather than duplicates. Check the notification-channel caveat above first. -
Test rollback: make a trivial change (e.g. edit
prompts/tasks/example_task.md), commit it,make deploy-devagain so a second version exists, then:make rollback REF=<first commit's short sha> ENV=dev make health-check
Confirm it checks out the ref, redeploys, and restores your original branch afterwards — including after a deliberately-broken intermediate commit, to prove the branch restore happens on failure too — and that
health-checkpasses against the rolled-back version.
The shared service account and staging bucket must survive. Deleting either breaks every other agent in the project. Deleting the shared SA or bucket during teardown is a documented past mistake — don't repeat it.
# 1. this agent's Agent Engine resource (the billable one).
# gcloud has no `ai reasoning-engines` command in some CLI versions — use the SDK.
uv run python -c "
import vertexai
from vertexai import agent_engines
vertexai.init(project='$ADK_TEST_PROJECT', location='europe-west1')
agent_engines.delete('$(cat .agent_engine_resource)', force=True)
print('deleted')
"
# 2. only this agent's staging subfolder
gsutil -m rm -r "gs://$ADK_TEST_PROJECT-agent-staging/deploy-test-agent/"
# 3. this agent's dashboard and two alert policies, by display name (Console or gcloud)- Do not run
gcloud iam service-accounts delete agent-engine-sa@... - Do not run
gsutil rm -ragainst the bucket root - Confirm any sibling agent's subfolder is still present afterwards
- Don't leave a
.envwith real values, or any SA key, behind in/tmp/agent-deploy
If instead you used a genuinely throwaway project, gcloud projects delete <project> removes
everything in one step and is the safer teardown — nothing shared, nothing to preserve.
This is a migration, not a fresh build — the goal is zero behavior change, just a new shell around the existing agent. Do this in a separate branch of the new generated repo, not inside this template repo.
- Generate the new repo first, empty, exactly as in §1c, and get
make devworking with the stock example tool before touching anything — confirms the scaffold itself is sound before you start pouring in existing code. - Inventory the source agent before moving anything:
- What ADK constructs does it use — plain
Agent(),sub_agents=[...], custom tool wrappers, custom callbacks? This template assumes "no custom classes, no inheritance — pure ADK syntax only" (per generatedCLAUDE.md); flag anywhere the existing agent deviates, since that's extra migration work, not a drop-in. - Which model provider does it currently use, and does it match one of
google/anthropic/openai/litellmindeployment/config.py? - Are prompts already externalized to files, or inline strings that need
extracting into
prompts/system/andprompts/tasks/plus aprompts.yamlregistry entry?
- What ADK constructs does it use — plain
- Port tools into
agent/tools/: one file per logical group, each function type-annotated and docstring'd (ADK builds the tool schema from these), wrapped in@instrument, re-exported fromagent/tools/__init__.py, and wired intotools=[...]inagent/agent.py. Write/port unit tests intotests/unit/test_tools.pyfor each as you go — don't batch this at the end. - Port prompts into
prompts/system/*.mdandprompts/tasks/*.md, register each inprompts/prompts.yaml. Diff the effective concatenated prompt (load_prompt()output) against the original agent's system prompt string — a line dropped during copy is a silent behavior change, not an error. - Port or write eval cases: add the existing agent's known-good and known-bad
examples into
tests/evals/datasets/skills/*.jsonl(or a new file, registered inpromptfoo.yaml). If the source agent has no eval suite today, this is the point where it gets one — don't skip it just because it's new work. - Re-run the full gate: §1c's
make lint && make typecheck && make test && make evallocally, then §2a's manualmake devconversation check, focused specifically on the behaviors the old agent was known for (its trickiest tool call, its safety edge cases). - Deploy once by hand per §3, against a genuinely separate dev GCP project/resource —
not the old agent's production Agent Engine resource — and compare its
make evalpass rate and a handful of manual conversations side-by-side against the old deployment before ever pointing real traffic at the new one. - Only after 6 and 7 both pass: retire the old deployment, on a timeline your team is comfortable with (keep it live in parallel for a while rather than a hard cutover).