Protect ProjectData storage with tool payload cleanup

[simple-agent-manager]

Summary

• Adds an automated ProjectData storage firebreak that strips expandable chat_messages.tool_metadata.content payloads from old terminal sessions when ctx.storage.sql.databaseSize crosses configurable pressure thresholds.
• Fixes the independent-review blocker: cleanup is now byte/memory-bounded, not just row-bounded. Candidate selection reads row identity plus length(CAST(tool_metadata AS BLOB)); full legacy metadata is read only through a per-row length(...) <= remainingBudget guard capped by PROJECT_DATA_TOOL_PAYLOAD_CLEANUP_BATCH_BYTES.
• Handles individual oversized legacy rows without reading them into JS by replacing the affected old terminal-session metadata with a small fail-closed sentinel and advancing the keyset cursor.
• Fixes poison-candidate/recheck behavior. Candidate strip/update failures are isolated per row, recorded through storageSafetyLastError/telemetry/logging, and cannot leave a stale past-due storageSafetyToolCleanupRecheckAt hot-looping the ProjectData alarm.
• Keeps high-value history: no chat rows are deleted, message text remains intact, active/sleeping sessions are excluded, and recent terminal sessions are protected by a configurable age floor.
• Fixes ProjectData storage measurement cadence so unrelated ProjectData alarms do not upsert storage telemetry before PROJECT_DATA_STORAGE_MEASURE_INTERVAL_MS is due.
• Adds the production storage-full literal (Exceeded the maximum database size.) to non-retryable Durable Object storage-full classification.

Candidate volume / control-loop cost: cleanup candidate selection is limited to terminal sessions (stopped, failed) older than PROJECT_DATA_TOOL_PAYLOAD_CLEANUP_MIN_SESSION_AGE_DAYS. Worst case per alarm is bounded by PROJECT_DATA_TOOL_PAYLOAD_CLEANUP_MAX_SESSIONS_PER_ALARM, PROJECT_DATA_TOOL_PAYLOAD_CLEANUP_BATCH_ROWS, and PROJECT_DATA_TOOL_PAYLOAD_CLEANUP_BATCH_BYTES, with raw cursors consumed before updates or awaits and a 60s default recheck delay when more candidates remain.

Staging/merge lane update (2026-08-25): final validation passed on head f521ad444a77a68517c8a9ddc74e6a86c0e096e2. The prior draft/needs-human-review hold was cleared by the independent review task, CI unblock task, focused local validation, and staging deployment/smoke evidence recorded below.

Validation

• pnpm --filter @simple-agent-manager/shared build
• pnpm --filter @simple-agent-manager/providers build
• pnpm --filter @simple-agent-manager/cloud-init build
• pnpm --filter @simple-agent-manager/api typecheck
• pnpm --filter @simple-agent-manager/api build
• (cd apps/api && pnpm vitest run --config vitest.workers.config.ts tests/workers/project-data-storage-safety.test.ts --reporter verbose) — 11 tests passed, including cumulative byte budget, oversized legacy row, poison row, stale recheck, and non-thrashing alarm scheduling coverage.
• pnpm --filter @simple-agent-manager/api test -- tests/unit/services/durable-object-retry.test.ts — 10 tests passed.
• pnpm --filter @simple-agent-manager/api test -- tests/unit/durable-objects/project-data-messages.test.ts — 10 tests passed.
• pnpm --filter @simple-agent-manager/api lint
• pnpm quality:file-sizes
• pnpm quality:ast-checks — 0 errors; pre-existing warnings remain unrelated.
• pnpm lint — passed with pre-existing warnings.
• pnpm format:check
• git diff --check

Staging Verification

• Staging deployment green — deploy-staging.yml run 32834218242 succeeded on head f521ad444a77a68517c8a9ddc74e6a86c0e096e2.
• Staging smoke tests passed — smoke-tests job 97763084122 succeeded; direct staging /health returned healthy at 2026-08-25T10:08:18.335Z. No UI-specific manual Playwright flow was required because this PR has no UI change.
• Infrastructure verification completed — N/A: no infra changes.
• Mobile and desktop verification notes added for UI changes — N/A: no UI changes.

Staging Verification Evidence

• deploy-staging.yml run 32834218242 succeeded on head f521ad444a77a68517c8a9ddc74e6a86c0e096e2.
• Jobs: Validate Configuration 97759393443 success; Deploy to Cloudflare 97759414284 success; smoke-tests 97763084122 success.
• Direct staging health: https://api.sammy.party/health returned status: healthy at 2026-08-25T10:08:18.335Z.
• Staging cleanup/capacity check after validation: staging D1 showed nodes only deleted=195 and workspaces only deleted=198; this validation created no VM nodes/workspaces requiring cleanup.

End-to-End Verification

• ProjectData alarm path traced from ProjectData.alarm() to runProjectDataStorageSafetyAlarm() to byte-bounded tool-payload cleanup.
• Worker-runtime tests exercise the ProjectData DO + SQLite storage + D1 telemetry path.
• Data safety assertions cover active/sleeping/recent-session preservation and chat message text preservation.
• Failure-path tests prove oversized and poison legacy metadata cannot stall cursor progress or re-arm stale past-due alarms.

Data Flow Trace

1. ProjectData.alarm() calls runProjectDataStorageSafetyAlarm() inside an isolated try/catch before unrelated ProjectData alarm work continues.
2. runProjectDataStorageSafetyAlarm() gates storage measurement through shouldMeasureProjectDataStorage() and then calls runProjectDataToolPayloadCleanup().
3. runProjectDataToolPayloadCleanup() reads sql.databaseSize, compares configurable trigger/target bytes, reads persisted cursor/recheck state, and selects only old terminal sessions.
4. selectToolPayloadCandidates() in tool-payload-cleanup-candidates.ts selects row identity plus length(CAST(tool_metadata AS BLOB)); it does not select full legacy metadata batches.
5. readBoundedToolMetadata() reads full metadata only for one candidate at a time and only when SQLite verifies the row fits the remaining byte budget.
6. Oversized or poison rows are fail-closed to small sentinel metadata, failure observability is recorded, and cursor/recheck state is advanced or cleared before the alarm is recalculated.

Untested Gaps

No staging verification by explicit task instruction. GitHub Actions were re-read after push and were pending on the new head at that time.

Post-Mortem

What broke

ProjectData storage growth approached Cloudflare’s 10 GB SQLite-backed Durable Object ceiling. The first PR #1901 implementation added automated cleanup but selected and materialized full legacy tool_metadata batches and could stall on a poison candidate.

Root cause

Large legacy tool metadata payloads accumulated in chat_messages.tool_metadata. Row-count-only cleanup is not memory-safe in the shared 128 MB Worker isolate, and exception handling around candidate processing was not isolated enough to guarantee cursor/recheck progress.

Class of bug

Storage-pressure control-loop gap / retention firebreak memory-bound and poison-candidate failure mode.

Process fix included in this PR

Task archive now records the independent-review blockers, the byte-budget/fail-closed fixes, focused validation commands, and specialist review evidence.

Post-mortem file

tasks/archive/2026-08-24-projectdata-storage-protection.md

Specialist Review Evidence

All listed specialist reviewers completed with PASS. Subsequent independent review task 01M0VXKZTQN9W84S9X60MT78Q3 reported CLEAR, CI unblock task 01M0W28FS0GDPH1KD3ZZ4N8XAW reported CI_UNBLOCKED on the current head, and final shipment-lane validation/staging passed.

Reviewer	Status	Outcome
cloudflare-specialist	PASS	Cleanup uses ctx.storage.sql.databaseSize; no runtime PRAGMA/VACUUM/auto_vacuum sizing path; candidate queries use raw(), row identity, and metadata byte length; full reads are per-row byte-guarded; no cursor crosses an await.
constitution-validator	PASS	New byte budget and existing thresholds/limits/cadence are configurable via env-backed defaults and documented in Env types, wrangler.toml, examples, public config docs, and env-reference.
test-engineer	PASS	Worker-runtime tests cover cumulative byte bounding, oversized single-row progress, poison-row progress, stale due recheck clearing, non-thrashing future alarm scheduling, and existing preservation/telemetry behavior.
task-completion-validator	PASS	Research findings and acceptance criteria map to code diff and tests; UI-to-backend and multi-resource checks are N/A.

Exceptions

No open merge-blocking exceptions after final validation/staging lane. The earlier staging/merge/readiness exception expired when the separate independent review reported CLEAR and this shipment lane completed local validation, CI verification, staging deployment, smoke tests, health check, and cleanup verification.

Agent Preflight

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

• SAM MCP task instructions for task/branch context.
• PR Protect ProjectData storage with tool payload cleanup #1901 independent review comment: Protect ProjectData storage with tool payload cleanup #1901 (comment)

Codebase Impact Analysis

• apps/api/src/durable-objects/project-data/storage-safety.ts: cleanup byte-budget config and alarm result flow.
• apps/api/src/durable-objects/project-data/tool-payload-cleanup.ts: cleanup orchestration, cursor/recheck persistence, telemetry/failure handling.
• apps/api/src/durable-objects/project-data/tool-payload-cleanup-candidates.ts: raw candidate selection, byte-bounded reads, oversized/poison fail-closed candidate processing.
• apps/api/src/durable-objects/project-data/tool-metadata-storage.ts: cleanup strip helper now reports parse/validation failure.
• Env/docs: Worker Env types, wrangler.toml, examples, public configuration docs, env-reference.
• Tests: Worker-runtime ProjectData storage-safety tests and related ProjectData/durable-object retry unit tests.

Documentation & Specs

• Updated apps/www/src/content/docs/docs/reference/configuration.md.
• Updated .claude/skills/env-reference/SKILL.md.
• Updated archived task record with validation/specialist-review notes.

Constitution & Risk Check

Checked Principle XI/no hardcoded values: byte budget, cleanup thresholds, target ratios, batch rows, max sessions, age floor, recheck cadence, and tool metadata cap all have env overrides. Main risk is data-loss perception for oversized/poison legacy metadata; mitigated by limiting scope to old terminal tool metadata under storage pressure, preserving chat rows/message text, and fail-closing only affected metadata rows.

[codspeed-hq]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing sam/implement-focused-projectdata-storage-yxeehd (f521ad4) with main (0e655f4)}

[simple-agent-manager]

Independent ProjectData storage-firebreak review: hold PR #1901. Do not advance out of draft / needs-human-review yet.

What looks correct:

• This is a real automated firebreak, not just telemetry: ProjectData.alarm() now calls runProjectDataStorageSafetyAlarm(), which can strip old terminal-session chat_messages.tool_metadata.content under storage pressure.
• Quota measurement uses sql.databaseSize; I found no runtime PRAGMA sizing, VACUUM, or auto_vacuum path in the cleanup.
• New thresholds/limits are configurable through Env/wrangler/env examples/public config docs.
• Retention is scoped to chat_sessions.status IN ('stopped','failed') and a configurable age floor; it updates chat_messages.tool_metadata only, not chat rows or message text.
• Local focused checks passed after building prerequisites: Worker-runtime ProjectData storage-safety test (8 tests), durable-object retry unit test (10 tests), and API typecheck. CI is green except the expected Specialist Review Evidence label gate.

Blocking defects:

1. Cleanup is row-bounded but not byte/memory-bounded for the legacy payloads it is meant to clean. selectToolPayloadCandidates() selects full tool_metadata values and parseToolPayloadCandidateRows() materializes up to PROJECT_DATA_TOOL_PAYLOAD_CLEANUP_BATCH_ROWS candidates into an array before stripping. With the default 500-row batch, this can load hundreds of legacy unbounded tool JSON payloads into the 128MB shared Worker isolate. The hard requirement was bounded cleanup that avoids the large-materialization/OOM class; row count alone is not a safe bound for the dominant legacy payload class. Add a byte budget or process much smaller byte-bounded pages, and add a test with large legacy payload rows proving the batch cannot exceed the memory budget.

2. Poison-candidate/failure behavior can re-arm forever. The cleanup loop has no per-row try/catch and no cursor advancement/clear-on-failure. If stripping/updating a legacy row throws while a persisted cleanup recheck is due, the old storageSafetyToolCleanupRecheckAt can remain in do_meta; computeStorageSafetyAlarmTime() will continue returning that past recheck time, and recalculateAlarm() can immediately re-arm the DO. Fix by isolating per-row cleanup failures, recording META_LAST_ERROR/structured log context, advancing past or quarantining the failed row without stripping it, and proving with a Worker-runtime test that malformed/poison tool metadata cannot cause immediate alarm thrash.

Keep the PR held until those are fixed and revalidated. Staging was intentionally not used per task instruction.

[simple-agent-manager]

Temporary CI refresh after removing needs-human-review: closing and reopening to trigger a fresh pull_request/reopened run on the same head SHA; no code changes.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.9% Duplication on New Code

See analysis details on SonarQube Cloud