Skip to content

feat(cdp): skip sandbox sends of a blocked organization - #587

Draft
Silthus wants to merge 21 commits into
workflows-sandbox/org-blockfrom
workflows-sandbox/org-block-worker
Draft

Silthus wants to merge 21 commits into
workflows-sandbox/org-blockfrom
workflows-sandbox/org-block-worker

Conversation

@Silthus

@Silthus Silthus commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Problem

A staff block from #586 stops new sandbox senders, but an organization's existing workflows keep sending sandbox email.

Stack: V3, protect shared reputation: #607 → #608 → #609 → #610 → #586 → this PR. #586 ships the block table this worker reads.

This fork PR moves to an upstream PostHog/posthog PR after review.

Impact

Pays into Grow Workflows through activation rate (measured here). In scope: Messaging, email sending. Expected move: small, up, as a rollout guard for the sandbox sender. Run cost: n/a.

Note

🟢 Merge risk: low · two-way door (merge-risk rubric)

  • Only sandbox sends read the block. The sandbox sender stays off by default behind WORKFLOWS_SANDBOX_SENDER_ENABLED and the workflows-sandbox-sender flag.
  • The organization member lookup gains a second query on the block table, run beside the member query. When it fails, sandbox sends skip; own-sender sends never run it.

Changes

  • A blocked organization's sandbox sends skip. The run log says: "Skipping send: the sandbox sender is turned off for this organization. Contact support to find out why."
  • The blocked event carries its own reason, organization_blocked, not recipient_not_member.
  • Own-sender sends ignore the block.
  • A block, and its removal, reach the email workers within one minute. The worker reads the block into the same one-minute snapshot as the organization members, and one call decides both.
  • reload-sandbox-sender-state refreshes only the shared tenant pause, so it cannot make a block take effect at once. The one-minute snapshot covers the block.
  • The block check runs first inside the recipient check. A blocked organization spends no daily caps, and the final check before delivery also blocks.
  • Mechanical: the reason union, and the members unit tests now count two queries per lookup.

How did you test this code?

Test rationale: email.service.test.ts, "a running worker stops sandbox sends within a minute of a block and resumes within a minute of its removal", for stored and test runs. One warm service and a controlled clock send, block, skip, unblock and send again. It catches a block that is ignored, folded into another reason, cached past the minute, or never lifted. Red before the change: the blocked send went out. Raising the snapshot age to an hour makes it fail.

Live run at 53833496c73. The real CdpApi app ran stored workflow test runs against a lane test database and a local SES fake. One block row, written as the admin writes it. On this base a skipped email step moves the flow on to its exit, so the run reports success. The skip shows as zero SES requests, the log line, and the event.

Case Block Worker Result SES requests
a none new sent 1
b added new skipped, the message above, event reason organization_blocked 0
c added new, own sender sent 1
d added cached lookup, under one minute old sent 1
e added same worker, 61 s later skipped, same message 0
f deleted new sent 1
g deleted the worker that saw the block, over a minute later sent 1

Case e sends no second event, because blocked events go out once per team, reason and mode per hour.

Cross-version. Django writes the block row and the email worker reads it.

Direction Sandbox send Own-sender send
Base worker (#586, the same worker code as #610) reads a database with a block row Sent. The old worker ignores the block. Sent
This worker reads a database without the block table Skipped as a failed member check Sent

Both outcomes are accepted. A block takes effect once the workers run this code. The table ships in #586, beneath this PR.

Gaps.

  • Prod shape: not measured. The block table is empty in production, so every team takes the not-blocked branch at rollout. Read-only production access arrives on 2026-11-02.
  • CI: the repo's full CI cannot run on this fork. Depot runner jobs never start, the Semgrep detection job is skipped, and the lane check counts the stack's lower layers. Local substitutes are listed below.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

docs/internal/workflows-email-sending-tiers.md says a blocked organization's sandbox sends skip, with the reason organization_blocked and a one-minute reach. #586 documents the admin and the endpoint.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

  • Skills: impact, tdd, writing-user-facing-copy, writing-pr-descriptions, pr-description.
  • Restack: feat(workflows): record sandbox deliveries as their own metric #610 restructured the member check, so the change was rebuilt test-first on it. The head merges feat(workflows): let staff turn off the sandbox sender per organization #586's branch into this branch without a force-push, so the block table is now in this PR's history. The merge keeps only this PR's change on top of the new base.
  • Proofs: fleet proof holds the live run and the cross-version run at 53833496c73, plus the prod-shape gap.
  • Review: GPT-6.1 Sol at high effort through fleet proof review. Round 1, on the old stack, found that the Node test only exercised cold caches. The warm-worker test above carries that fix. Round 2 read the PR's diff against feat(workflows): record sandbox deliveries as their own metric #610. It found that the block check and the member check read the snapshot separately, so a refresh between them could let a blocked send through. c30c802a210 decides both from one snapshot. The final read on c30c802a210 found nothing, and so did the final reads after the V3 restack, on fcc109f116e and 53833496c73.
  • Lane gate, green locally on this head's tree: the email service, organization members and sandbox email sender suites, the rate limiter suite, oxlint, eslint and prettier on the changed files, and the pre-push hogli ci:preflight --strict.
  • Typecheck: tsc --noEmit on fcc109f116e reports errors only from a native module this box does not build, in files this PR does not touch.
  • fleet proof gate: passes on 53833496c73, with the CI gap accepted.
  • No duplicate: the change builds on this fork's sandbox sender stack, which no other PR covers.
  • Public artifact: every fixture and address is invented.
Question Answer Who Where
Where do the block and its worker sit in the stack? In V3, protect shared reputation: after the sandbox metric (#610), with #587 right above #586 👤 Human comment
How do staff turn off the sandbox sender for one organization? A staff-managed block table in the Django admin, its own reason and message, and no new senders. No customer UI. 👤 Human comment
Accept the CI gap for this fork PR? Yes: no depot job runs on the fork, and the local lane gate covers the changed code 👍 Accepted default comment
Is a worker that reads the database before the migration acceptable? Yes, as a landing order: #586 now sits beneath this PR, and migrations run before workers deploy 👍 Accepted default comment
How fast does a block reach the workers? Within one minute, through the organization member snapshot. The reload message only covers the tenant pause. 🤖 Agent comment
Where does the send path check the block? First inside the recipient check, so caps stay unspent and the final check also blocks 🤖 Agent comment
How does the worker read the block? A second query beside the member query, in the same snapshot, and one call returns both decisions. One joined query would change the member query's semantics. 🤖 Agent comment

Involvement: no grilling rounds, two cross-model review rounds and four final reads, proof captured.

🤖 Generated with Claude Code

Silthus and others added 2 commits October 26, 2026 08:11
The email worker reads SandboxSenderOrganizationBlock in the team
organization lookup. While a block row exists, the organization's sandbox
sends skip with the reason organization_blocked and a run-log message that
points to support. The check runs inside the recipient check, so a blocked
organization spends no daily caps and the final check before sending also
blocks. A block reaches the workers within one minute. Own-sender sends do
not read it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus force-pushed the workflows-sandbox/org-block-worker branch from 86359b8 to 4e16aa2 Compare October 26, 2026 08:14
Silthus and others added 15 commits October 27, 2026 05:43
A send through a `sandbox` email integration now skips unless the worker
switch and its three SES route settings are set, and unless every To, CC
and BCC address belongs to an active, verified member of the team's
active organization. The check runs again right before delivery. Each
skip logs a reason and emits a deduplicated blocked event. The sandbox
route does not exist yet, so a send that passes every check still skips
as unavailable instead of reporting success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sandbox send reaches only the team's own members, so it must not spend or wait on the
team's tier budget that its real sends share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sandbox send that passes the member gate now goes out through the fixed
SES tenant, configuration set and From address, with a footer that names
the sender and the organization. Tracking, unsubscribe headers, the
team's tier budget and custom From/Reply-To settings do not apply. The
check on the final recipient list runs right before delivery, and each
delivered send emits a sandbox sent event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ier history

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sandbox sends claim a daily allowance per project and per recipient
address across the organization right before delivery. A reached
allowance skips the send. SES throttles give the claim back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CDP API's SES Valkey pool no longer stops the process on repeated
errors, and it bounds connection and command waits, so an outage skips
sandbox sends instead. A claim now carries its own return deadline, so
an old claim cannot refund into a refilled bucket.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Django records the pause state of the shared sandbox SES tenant in
SandboxSenderTenantState, from the tenant-state webhook and the daily
reconciliation, and announces each committed change to the workers.
Nothing reads the table yet; the worker gate follows in its own PR.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…used

The email worker reads the sandbox SES tenant state that Django mirrors,
skips every sandbox send while the tenant is paused or the state cannot
be read, and reloads the state on the reload-sandbox-sender-state message.
After the daily allowance claim it checks the pause again and gives the
claim back when the pause blocks delivery.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tier sweep subtracts email_sandbox_sent from email_sent per day, and
the sending allowance subtracts it per workflow, both floored at zero.
No code records email_sandbox_sent yet, so the tier does not change until
the worker metric lands in its own PR.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A successful sandbox send records email_sandbox_sent next to email_sent,
so the sending tier can leave it out while the workflow metrics still
show it. The CDP API's SES webhook captures a workflows sandbox email
delivered event for deliveries on the sandbox configuration set.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…subtraction

The allowance case now gives one workflow more sandbox sends than sends,
so a team-wide subtraction would erase the other workflow's own sends.

Refs Silthus/personal#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dbox stack

Merge the new base, workflows-sandbox-delivery/sandbox-metric, into this branch, and keep only this PR's change on top of it. The worker reads the block with the organization members.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus changed the base branch from workflows-sandbox/org-block to workflows-sandbox-delivery/sandbox-metric October 27, 2026 12:31
…shot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decisions for this PR:

  • 👤 Where do the block and its worker sit in the stack? In V3, protect shared reputation: after the sandbox metric (feat(workflows): record sandbox deliveries as their own metric #610), with feat(cdp): skip sandbox sends of a blocked organization #587 right above feat(workflows): let staff turn off the sandbox sender per organization #586.
  • 👤 How do staff turn off the sandbox sender for one organization? A staff-managed block table in the Django admin, with its own reason and message, and no new senders. No customer UI.
  • 🤖 How fast does a block reach the workers? Within one minute, through the organization member snapshot. The reload message only covers the tenant pause.
  • 🤖 Where does the send path check the block? First inside the recipient check, so daily caps stay unspent and the final check before delivery also blocks.
  • 🤖 How does the worker read the block? A second query beside the member query, in the same snapshot, and one call returns both decisions. One joined query would change the member query's semantics.
  • 👍 Accept the CI gap for this fork PR? Yes. No depot job runs on the fork, and the local lane gate covers the changed code.
  • 👍 Is a worker that reads the database before the migration acceptable? Yes, as a landing order. feat(workflows): let staff turn off the sandbox sender per organization #586 now sits beneath this PR, and migrations run before workers deploy. A worker without the table skips sandbox sends, and own-sender sends are unaffected. The lane rule keeps the migration out of this PR.

@Silthus
Silthus force-pushed the workflows-sandbox-delivery/sandbox-metric branch from 69afe3e to 7e95863 Compare October 28, 2026 10:00
…ock layer

Merge workflows-sandbox/org-block (#586) without a force-push.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus changed the base branch from workflows-sandbox-delivery/sandbox-metric to workflows-sandbox/org-block October 28, 2026 11:00
Silthus and others added 2 commits October 28, 2026 11:06
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The block layer documents the admin and the endpoint, so this layer only adds the worker side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant