[Silthus]

Moved here from #121897, closed on 2026-10-25 so this fork stack is the only copy. It opens upstream again after review.

Problem

• A caller of the transactional email API cannot see what happened to an email after the 202: queued, sent, delivered, bounced or skipped. Contract #120921 promises a status read by message_id.
• This layer writes one status row per change into the table from #121896. The read route is #121898.
• Map: Silthus/posthog#461, slice 08 of strand Silthus/posthog#535, layer 2 of 3.

Impact

Pays into Grow Workflows through billed workflows_emails actions per paying org (not measured yet as a monthly count). In scope: Messaging, email API sends. Expected move: small, up: a caller can follow an email to delivery, as other providers' APIs allow. Run cost: one Kafka produce per status change of a transactional email.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door: the new run field and SES tag are optional, so a revert leaves old jobs and tags harmless (merge-risk rubric)

• The email worker and the SES webhook now wait for Kafka on a transactional email. A failed write in the worker never fails or resends the email; it is counted. A failed write in the webhook answers SNS with a 5xx, so SNS retries.
• The run state gains emailRecipientHash, a queued format. Readers on both sides tolerate its absence and its presence, below.
• Only sends through the transactional email route write rows, and they sit behind the workflows-transactional-email-api flag. Rollout precondition: the flag stays off until tracking code v3 minting is on and every stack PR has deployed on Django, cdp-api, the hogflow worker and the email worker (#121272 has the reason).

Changes

Kind	Files	Lines
Production	21	+518 −22
Tests	11	+653 −9
Total (this layer)	32	+1171 −31

What each part writes

Where	Status	reason	Waits for Kafka
cdp-api's transactional email route, once the run is queued	queued		No
The email worker	sent, failed, skipped	provider_error, or the send check's code such as template_paused	Yes
The SES webhook	delivered, bounced, complained, failed	hard, soft or unknown for a bounce; the feedback type for a complaint; rejected for an SES Reject	Yes, 5xx to SNS on failure
The janitor	unknown when it resets a stalled email job, failed when it gives up on one	gave_up	Yes
The email action, when it skips before the email service runs	skipped	recipient_suppressed, recipient_opted_out, invalid_recipient or frequency_capped	Yes
The hog flow worker, when it cancels a run whose workflow stopped, a run whose cancel was requested, or a run whose workflow or team it cannot load	skipped	workflow_inactive, run_canceled, workflow_missing	Yes

How it works

• EmailStatusService (email-status.service.ts) implements EmailStatusWriter with record (awaited) and recordInBackground. It produces to clickhouse_email_status, keyed by the message id.
• A status only moves forward, by rank: queued, unknown, skipped, sent, failed, delivered, bounced, complained. failed ranks above sent because SES can reject an email it accepted.
• A failed attempt (failed with provider_error or gave_up) ranks with skipped. A rerun keeps the message id, so its outcome replaces the attempt it reruns: between a skip and a failed attempt the later one wins, and sent replaces both.
• The message id is the run's invocation id, a UUIDT. Each row's created_at comes from its time, so every row of a message carries the same created_at.
• cdp-api hashes to once (emailRecipientHash: HMAC-SHA256 with a key derived from the first ENCRYPTION_SALT_KEYS entry, over the trimmed, lowercased address) and stores it on the run as emailRecipientHash.
• The worker adds the hash to a transactional send as the SES tag ph_recipient_hash. The webhook writes a status only for a record with that tag and a valid tracking code, so other workflow emails write nothing.
• A delivery, bounce or complaint moves the status only when one of its recipients hashes to the tag, under any salt key. So a cc or bcc copy that bounces leaves the status of to alone.
• A run counts as a transactional email when its trigger event is $transactional_email.
• A rerun keeps emailRecipientHash, so its status rows are still written.
• No row, log line, metric label or error message carries the address.
• Without a usable salt key, the route still queues and answers 202. Only the status write is skipped, and cdp_email_status_missing_recipient_hash{source="route"} counts it, once per email the route queues.
• The janitor's stalled-job reset returns no job state. It then reads the state of stalled email jobs in chunks of 100 and writes their unknown rows in parallel per chunk.
• Metrics: cdp_email_status_rows_produced{status}, cdp_email_status_write_failed{status}, cdp_email_status_missing_recipient_hash{source}.

How did you test this code?

Test rationale: each case below failed before its change and passes after it.

Test seams:

• EmailStatusWriter at the Kafka boundary, with the outputs faked: one row per status, its columns, and a later-ranked status at an earlier time keeps the higher version.
• The recipient hash, with a fixed vector that the read layer asserts on the Django side: salt key 00beef0000beef0000beef0000beef00, " Jane.Doe@Example.COM " and "jane.doe@example.com" both give e7b76d6c58afac8ef397725c6b81a97de6e27926a51b37ee79b281032c3f2aa7.
• Each write point at its nearest existing test. These tests run the real EmailStatusService and read the rows at the Kafka producer, with UUIDT message ids the writer accepts:
  • cdp-api's serial suite: an accepted send writes queued; a replay and a failed queue write none; with no salt key, a send answers 202, queues once and writes no status.
  • The email service: sent, failed with provider_error, skipped with the send check's code; a failing status write still sends once; a run without the hash sends once, writes nothing and is counted.
  • The SES webhook: delivered, bounced and complained rows, no row for a not-spam correction of a complaint, and a 5xx when the write fails.
  • The SES webhook: a Reject after sent reads failed with rejected, and a bounce of a bcc copy after delivery to to keeps delivered.
  • The hog flow worker: a transactional run whose workflow stopped, whose cancel was requested, or whose workflow is gone writes skipped with workflow_inactive, run_canceled or workflow_missing.
  • The janitor: unknown and gave_up, and more stalled email jobs than one chunk each get one unknown row.
  • The writer: a rerun replaces an attempt that failed with provider_error or gave_up by one that is sent or skipped, and the other way round.

Local gate at the stack top 987b927630e, which holds this head unchanged, all green: cdp-api serial 288, CDP jest 3690 (1 skipped), nodejs tsc, and the rest of the gate. The pre-push preflight ran in strict mode with 0 failures.

Cross-version: owed, because the run state gains emailRecipientHash and the send gains a tag. A backward-compatible reader covers both directions:

Producer	Consumer	Result
New cdp-api, run with emailRecipientHash	Old worker	Ignores the field. The queue reads the state with a plain parseJSON and copies it whole (job-queue-postgres-v2.ts:346, :361; job-queue-kafka.ts:187). No schema validates it.
Old cdp-api, run without it	New worker	Sends once, writes no status, counts source="worker". A test pins it.
New worker, tag ph_recipient_hash	Old webhook	Ignores the tag.
Old worker, no tag	New webhook	Writes no status.

Live run at 987b927630e (Django, cdp-api and the email worker on a migrated clone of the local test database, with ClickHouse and maildev), case 07: a code send read sent by its id two seconds after the send. A read by recipient listed the three accepted sends, each sent, and none of the refused ones (cases 04 and 05).

Earlier live run at 1e18413cd1c, before the review fixes. It shows the rows in ClickHouse, which the final run did not read:

GET messaging/emails/<message_id>/ every 2s until sent
t+2s HTTP 200 {"id":"01a166ba-6dbd-0001-c3b7-33e2dea5fe57","status":"sent","created_at":"2026-10-23T03:07:36.253000Z","status_at":"2026-10-23T03:07:36.738000Z"}
an unknown id: HTTP 404 {"code":"not_found", ...}
clickhouse_email_status: queued, then sent; email_status keeps sent with a 64-character recipient_hash

Gaps:

• The missing-hash counter counts once per worker attempt, so a retried run counts more than once.

Release status

• This change is behind a feature flag and is not available to users

Docs update

None. The transactional email API docs page is written after the stack lands.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

Question	Answer	Who	Where
Can emails be looked up by recipient?	Yes: every row carries an HMAC of the normalized address, never the address	👤 Human	pending approval
Which secret keys the recipient hash?	A key derived from the first ENCRYPTION_SALT_KEYS entry, which Node and Django both read, over a new secret	🤖 Agent	pending approval
What happens without a salt key?	The send goes out and only the status write is skipped and counted, over refusing the send	🤖 Agent	pending approval
How does the webhook know the hash?	From a send-time SES tag, over hashing the SES destination, so copies cannot change it	🤖 Agent	pending approval
Where does created_at come from?	The message id's UUIDT time, so the run carries only the hash	🤖 Agent	pending approval
How does a rerun's outcome replace a failed attempt under the same message id?	A failed attempt (provider_error, gave_up) ranks with skipped, so the later of the two wins, and sent replaces both. Side effects: a failed attempt never replaces sent, and an SES Reject (failed with rejected) still ranks above sent.	🤖 Agent	pending approval

No open questions in this layer. The rollout decision is on #121272.

Review (GPT-6.1 Sol, round 1 over the whole stack, then a final read of the fixes):

Finding	Response
A skip before the email service leaves the status at queued	Fixed: the email action and the hog flow worker record skipped with a reason. Tests: each skip in the action handler, and the worker's cancel.
A rerun drops emailRecipientHash	Fixed: the rerun restores it. The replay-fidelity test checks it.
Final read	No findings.

The stack's QA swarm then reviewed this layer. Round 1 fixed seven items, among them an SES Reject after sent (now failed with rejected) and a bounced cc or bcc copy that moved the status of to. Round 2 found that a rerun's sent read failed, and the fixer ranked failed attempts low. Final read 1 found they then ranked below skipped, and the fix ranks them with skipped. Final read 2 found nothing.

Involvement: strand conductor relays answered, one choice from Michael on replay scope, no grilling rounds. Reviews: GPT-6.1 Sol round 1 over the whole stack and a final read of the fixes. Stack QA swarm: design pass (GPT-6.1 Sol high); round 1 and round 2 (review and team lanes GPT-6.1 Sol high, fresh eyes Claude Opus 5.5 high); final reads 1 and 2 (GPT-6.1 Sol high); fixers Claude Opus 5.5 high; live run at the stack top.

• Skills invoked: /tdd, /writing-tests, /writing-code-comments, /writing-pr-descriptions.
• Public artifact: all fixtures are invented and use example.com domains.

🤖 Generated with Claude Code

[github-actions]

🤖 CI report

⚠️ Comment density — 3% of added code lines are comments (35 of 1041)

This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out.

Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See .agents/skills/writing-code-comments/SKILL.md for the house rules.

Files with the most added comment lines:

File	Comment lines	Added lines
nodejs/src/cdp/services/messaging/email-status.service.ts	13	136
nodejs/src/cdp/services/messaging/email-recipient-hash.ts	4	22
nodejs/src/cdp/services/messaging/helpers/ses.ts	4	75
nodejs/src/cdp/config.ts	2	7
nodejs/src/cdp/services/cyclotron-v2/janitor.ts	2	74
nodejs/src/cdp/services/messaging/email-tracking.service.ts	2	11
nodejs/src/cdp/services/messaging/email.service.ts	2	47
nodejs/src/cdp/cdp-api.serial.test.ts	1	50

This check does not block merging. It updates on every push and clears when the share drops.