Speedy, auditable, deadlock-resistant storage tracking #2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeremy
force-pushed
the
storage-ledger
branch
5 times, most recently
from
dd521fe to
2d2cc66
Compare
jorgemanrubia
approved these changes
Dec 9, 2025
Member
jorgemanrubia
left a comment
jorgemanrubia
left a comment
There was a problem hiding this comment.
Love it @jeremy. This is much better than the previous approahces. Thank you 👏👏
jeremy
force-pushed
the
storage-ledger
branch
13 times, most recently
from
4c6f4fc to
7115b2e
Compare
jeremy
force-pushed
the
storage-ledger
branch
2 times, most recently
from
0f69f4a to
873d5a8
Compare
An append-only storage ledger replaces deadlock-prone synchronous counter updates and drift-prone async updates. All content storage (card images, card/comment/board description embeds) is tracked. Account exports (which expire) and avatars are not tracked. Storage is accounted for by Account and Board. Both consume the same event stream independently: no bubble-up storage bumps triggering deadlocks due to lock sequencing. Each calculates its own total from the underlying ledger with independent cursors and materialization. * Storage::Entry: Append-only ledger recording attach/detach/transfer events with delta bytes. Single event stream indexed for both Account and Board cursor queries. * Storage::Total: Polymorphic snapshot cache with cursor (last_entry_id) tracking which entries have been materialized. * Storage::Totaled concern: Provides bytes_used (fast snapshot) and bytes_used_exact (snapshot + pending) query modes, plus materialize_storage! to roll up pending entries. * Storage::Tracked concern: For models owning attachments (Card, Comment, Board). Provides board_for_storage_tracking for models where board is determined differently (Board returns self). Handles board transfers by recording transfer_out/transfer_in entries. * Storage::AttachmentTracking: Hooks ActiveStorage::Attachment lifecycle to record attach/detach entries. Handles ActionText::RichText embeds by traversing to the actual model. Snapshots context in before_destroy to handle cascading deletes where parent record may be gone by after_destroy_commit. * MaterializeJob: Rolls up pending entries into snapshot. Concurrency limited per owner to prevent duplicate work. * ReconcileJob: On-demand reconciliation against actual attachment storage for support/debugging. Compares ledger total to real bytes from card images, card embeds, comment embeds, and board embeds. Usage: * account.bytes_used / board.bytes_used: fast, slightly stale bytesize * account.bytes_used_exact / board.bytes_used_exact: real-time bytesize * Storage::Entry: audit trail for debugging and point-in-time queries
jeremy
force-pushed
the
storage-ledger
branch
from
873d5a8 to
7fcf211
Compare
barturba
added a commit
to barturba/fizzy
that referenced
this pull request
Dec 11, 2025
Resolved merge conflicts in config/deploy.yml and config/environments/production.rb while preserving fork-specific functionality: - Adopted upstream's SMTP_ADDRESS naming (was SMTP_HOST) - Adopted upstream's conditional SMTP configuration pattern - Preserved fork's custom secrets: APP_HOST, REPLY_TO_EMAIL, ALLOWED_EMAIL_ADDRESSES - Preserved fork's default_url_options for multi-tenant URL generation - Updated .kamal/secrets.production to use SMTP_ADDRESS - Updated DEPLOYMENT.md documentation Upstream changes include: - Basic API functionality (basecamp#1766) - CSP enforcement (basecamp#2070) - Speedy storage tracking (basecamp#2026) - MySQL SSL configuration - Many bug fixes and improvements
adjogima
added a commit
that referenced
this pull request
Dec 11, 2025
* main: (117 commits) Explain that the upload URL is account-scope Allow direct uploads via API Storage: ignore jobs for now-deleted targets API: Support `created_at` for API card and comment creation (#2056) Enforce CSP (#2070) CSP: full config with env vars per source (#2069) Speedy, auditable, deadlock-resistant storage tracking (#2026) Gitleaks: ignore legit non-sensitive API keys and tokens in docs/ and test/ (#2068) Get gitleaks-audit green again Bump actions/checkout from 4 to 6 (#2047) Bump docker/login-action from 3.5.0 to 3.6.0 (#2046) Bump docker/metadata-action from 5.8.0 to 5.10.0 (#2045) Bump sigstore/cosign-installer from 3.9.2 to 4.0.0 (#2044) make MySQL SSL mode configurable via env var (#2036) Update tip text for turning a card into a Golden Ticket Revert "Fix Lexxy prompt list padding by lowering rich-text specificity" Fix Lexxy prompt list padding by lowering rich-text specificity Improve phrasing Fix crash due to missing partial Fix status and filter mistakes ...
adjogima
added a commit
that referenced
this pull request
Dec 11, 2025
…tylesheets+edits * mobile-app/scoped-stylesheets: (117 commits) Explain that the upload URL is account-scope Allow direct uploads via API Storage: ignore jobs for now-deleted targets API: Support `created_at` for API card and comment creation (#2056) Enforce CSP (#2070) CSP: full config with env vars per source (#2069) Speedy, auditable, deadlock-resistant storage tracking (#2026) Gitleaks: ignore legit non-sensitive API keys and tokens in docs/ and test/ (#2068) Get gitleaks-audit green again Bump actions/checkout from 4 to 6 (#2047) Bump docker/login-action from 3.5.0 to 3.6.0 (#2046) Bump docker/metadata-action from 5.8.0 to 5.10.0 (#2045) Bump sigstore/cosign-installer from 3.9.2 to 4.0.0 (#2044) make MySQL SSL mode configurable via env var (#2036) Update tip text for turning a card into a Golden Ticket Revert "Fix Lexxy prompt list padding by lowering rich-text specificity" Fix Lexxy prompt list padding by lowering rich-text specificity Improve phrasing Fix crash due to missing partial Fix status and filter mistakes ...
coorasse
reviewed
Dec 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
An append-only storage ledger replaces deadlock-prone synchronous counter updates and drift-prone async updates.
All content storage (card images, card/comment/board description embeds) is tracked. Account exports (which expire) and avatars are not tracked.
Storage is accounted for by Account and Board. Both consume the same event stream independently: no bubble-up storage bumps triggering deadlocks due to lock sequencing. Each calculates its own total from the underlying ledger with independent cursors and materialization.
Usage:
/cc @basecamp/sip