Skip to content

feat(cdp): send a template from the workflow version Django graded - #584

Draft
Silthus wants to merge 4 commits into
email-v2/09-route-idempotency-cdpfrom
email-v2/10c-template-version-on-send
Draft

Silthus wants to merge 4 commits into
email-v2/09-route-idempotency-cdpfrom
email-v2/10c-template-version-on-send

Conversation

@Silthus

@Silthus Silthus commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Part of the code-first rebuild of the email send stack; new in the rebuild, with no closed PR to replace: a template send delivers the content Django graded (decision D3 on map #461).

Problem

  • A template send on send/email/ right after an upload can deliver the previous content, while Django answered that the caller's template.hash is current.
  • Django grades the hash against Postgres. The email goes out from the workflow copy that cdp-api and the hogflow worker cached. A lost or late reload leaves that copy behind for up to two minutes.
  • This is the cdp-api and worker half. The Django half, which sends the version, is feat(messaging): send the workflow version a template send graded #585.

Impact

Pays into Grow Workflows through billed workflows_emails actions per paying org: a developer who uploads a template and sends it at once gets the new content, so the first send from code is right. In scope: CDP email sends.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door: a revert sends from the cached copy again, as today

Changes

Kind Files Lines
Production 5 +98 −12
Tests 3 +206 −0
  • A template send uses a workflow version at least as new as the one Django graded. The accepted send can carry hogFlowVersion. When the cached copy is older, cdp-api and the hogflow worker read the workflow from the primary.
  • One primary read per workflow serves a burst. Concurrent sends after one upload share one read, and its fresh copy goes into the cache. An older replica load cannot replace it.
  • If even the primary is older, nothing is sent. cdp-api queues nothing and answers Django with an internal 503. Django answers the caller with the contract's 503 delivery_unavailable, and puts the version lag in detail and why (feat(messaging): send the workflow version a template send graded #585).
  • The reason stays internal. workflow_version_behind appears only in the body cdp-api sends to Django, in the metric label and in the log. No public answer carries it.
  • A queued job in that state fails in the worker, with a log line that names both versions and a workflow_version_behind count. Versions only go up, so this state should not happen.
  • The version stays out of the idempotency fingerprint, so a keyed retry that Django grades after an upload still replays.
  • Why a version floor and not a hash key: Node has no copy of the content hash. The worker, not cdp-api, decides which content goes out. The template save writes the content and the bumped workflow version in one transaction. So "at least the graded version" means "the graded content or newer", and both sides already have version. An upload that lands between the grade and the send delivers the newer content. It is never older.
  • Limit: the floor covers versioned writes. The asset-URL rewrite command, as fix(messaging): sync the transactional workflow on asset url rewrites #571 reworks it, rewrites a transactional workflow's content before its template sync runs, so the workflow version doesn't bump.
  • Running the template pass first bumps the version. But then the real run reports fewer changed rows than the dry run, and its output doesn't say why. So the rewrite stays a known exception, and a send right after one can still deliver the cached copy.

How did you test this code?

Test seams: cdp-api's accepted-send route over HTTP (cdp-api.serial.test.ts), processInvocations on the hogflow worker, and the hogflow manager service for the read count.

Test rationale, each run red first:

  • A send whose graded version is newer than the cached copy that delivers the cached content, in cdp-api and in the worker.
  • A graded version newer than the database that queues, or a worker job that is not dequeued as failed.
  • A burst of sends after one lost reload that makes more than one primary read, or a later send that reads the primary again.
  • A 503 body from cdp-api that carries a public code: it stays the internal {error} shape of cdp-api's other 503s.
  • Passing before and after, pinning today's behavior: a matching version that reads the primary, a send without the field, and a keyed retry graded at a newer version that doesn't replay.

Proofs: live runs saved with fleet proof, curl into maildev. A lost reload is simulated in SQL: the workflow's version and subject change with no reload published.

Case Both new (2bdc74b, Django #585 4d76e98) Node at #580, Django #585 Django before #585, Node new cdp-api new, workers at #580
Send by key and hash, caches warm 202, delivered 202, delivered 202, delivered 202, delivered
Lost reload, then a send by key and hash 202, delivers the new content 202, delivers the cached content, as today 202, delivers the cached content, as today 202, delivers the cached content, as today
Lost reload, then a send by template id 202, delivers the new content not run not run not run
Keyed send before the lost reload, retried after it 202 replayed: true, one email in total not run not run not run
Five sends at once after a lost reload each 202 and delivers the new content; cdp-api makes one primary read not run not run not run
Rendered-content send 202, delivered 202, delivered 202, delivered 202, delivered
Template by id (control) 202, delivered 202, delivered 202, delivered 202, delivered
Version-behind 503s none – – –
  • The read count comes from cdp-api's Postgres query metric for primary reads of workflows. It rose by 1 across the five sends.

  • This head, ae46ab4, differs from 2bdc74b only by the 503 body. No case above reaches that 503.

  • Each mixed window delivers the cached copy, which is today's behavior: the older side drops or ignores the version.

  • Prod shape: a gap until read-only production access exists on 2026-11-02.

CI: the fork has no Depot runners, so those jobs never start, and no secret for the project-board app, so flags-project-board fails; the other GitHub-hosted checks pass. The lane check passes: against master the PR changes only nodejs/.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

Question Answer Who Where
Does a template send deliver what Django graded? Yes, through a new Node-chain PR after #580 👤 Human map #461, decision D3 of 2026-10-25
May the code carry comments? No; names carry the reasons 👤 Human map #461, 2026-10-25
What if even the primary is older? Send nothing and answer the contract's 503 delivery_unavailable with the version lag in detail and why; replaces a new public workflow_version_behind code 👤 Human approved (conductor, 2026-10-26)
Key the cache on the hash, or on a version floor? A version floor, because Node has no hash and the worker decides delivery; replaces D3's "key the cache on the hash" 👍 Accepted default approved (conductor, 2026-10-26)
Bump the workflow version on the asset-URL rewrite in #571? No; keep it a named known limit, because the reorder makes the dry run and the real run report different totals 👍 Accepted default approved (conductor, 2026-10-26)

Involvement: one decision from Michael and one from the map's conductor; review round 1 (GPT-6.1 Sol, Opus fresh eyes) fixed, then final reads by GPT-6.1 Sol; live and cross-version proof captured.

🤖 Generated with Claude Code

Silthus and others added 3 commits October 26, 2026 02:38
An accepted send can carry hogFlowVersion, the version of the template's
workflow Django graded the hash against. When the cached workflow is
older, cdp-api reads it from the primary. When the primary is older too,
it answers 503 workflow_version_behind and queues nothing. The version
stays out of the idempotency fingerprint, so a retry graded after an
upload still replays. Without the field, the send behaves as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hogflow worker reads the job's workflow from the primary when its
cached copy is older than the version the accepted send carries, and
fails the job when the primary is older too, so the worker never sends
older content than Django graded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Concurrent sends that found an older cached workflow each read the
primary, and the fresh copy was never cached, so the next send read the
replica and the primary again. Primary reads are now shared per
workflow and the result goes into the cache. A later replica load keeps
a newer cached version. A send that joins a read which cannot meet its
graded version reads the primary once more.

The worker logs a version-behind job with both versions and counts it
on cdp_hogflow_worker_jobs_failed_total, and the cdp-api refusal
carries code workflow_version_behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workflow_version_behind sub-code stays in logs, the outcome label and
the cdp-api-to-Django body. Django maps it onto its existing public 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant