object-store: provider-neutral seam + native Google Cloud Storage provider with generation CAS - #6762
Open
NateJowett wants to merge 7 commits into
Open
object-store: provider-neutral seam + native Google Cloud Storage provider with generation CAS#6762NateJowett wants to merge 7 commits into
NateJowett wants to merge 7 commits into
Conversation
NateJowett
force-pushed
the
mozart/gcs-object-store
branch
2 times, most recently
from
August 31, 2026 12:29
b9ccfb0 to
e16c8c7
Compare
🔐 Codex Security Review
|
NateJowett
marked this pull request as ready for review
August 31, 2026 12:32
Buzz keeps media blobs and the content-addressed Git object store on one bucket, but each held its own rust-s3 client and spoke in S3-shaped vocabulary: ETags, If-Match, If-None-Match: *. That made the storage provider a property of every call site rather than a property of the deployment, and left no place to add a second provider. Add `buzz-object-store`, which owns the seam: - `ObjectStore` — exactly the operations Buzz performs, no more: buffered and streaming upload, full/bounded/range/streaming read, head, create-only and revision-matched conditional writes, paginated prefix listing, single and bulk delete, connectivity and versioning admission checks. - `Revision` / `WriteCondition` / `ConditionalWrite` / `ImmutableWrite` — provider-safe types, so a compare-and-swap token is never a bare string with implied semantics. A revision minted by one provider cannot predicate a write against another; the accessor rejects it. - `ObjectStoreError` — an explicit taxonomy separating a *classified* provider answer (not found, precondition failed, throttled, transient, permanent) from an *unknown* transport outcome. That distinction is load bearing: the Git conformance probe drops unknown outcomes from its observer set rather than counting them as a lost race, so `TransportAmbiguous` stays reserved for pre-classification failures. - `ProviderKind`, so a deployment names its provider once. No behavior change: this commit adds the crate, its unit tests, and registers it in the workspace. The domain facades move onto it next. Signed-off-by: mozarthq <nate@mozarthq.com>
`MediaStorage` and `GitStore` were each a concrete `rust-s3` wrapper. They
built their own `Bucket`, matched on `S3Error`, and carried S3 vocabulary
through domain code. Move all of that into `providers::s3` — now the only
module in the tree that knows what an ETag is — and leave behind the parts
that are genuinely domain logic.
Media keeps its content-addressed blob keys, the community-scoped metadata
sidecar that gates tenant reads, and its error surface. `MediaStorage::new`
still takes a `MediaConfig` and builds the S3 provider, so every existing
caller is unchanged; `MediaStorage::with_store` wraps an already-constructed
provider and `object_store()` hands it out, which lets the relay build one
client per process instead of two against one bucket. `S3AddressingStyle`
moves to the object-store crate — it configures the provider, not the media
domain — and is re-exported from `buzz_media::config` so its parsing tests
are untouched.
Git keeps its content-addressed keys, digest verification, the idx sidecar
layout, and the conformance probe. `ETag` becomes `Revision`, `Precond`
becomes `WriteCondition`, and `CasOutcome` becomes `ConditionalWrite`.
`StoreError::Backend` wraps `ObjectStoreError`, and the `From` impl lifts
not-found / too-large / digest-mismatch into their own variants so every
existing call site keeps matching on them directly.
Probe semantics are unchanged. Its drop-and-floor rule — a racer that never
got a classified provider response is dropped from the observer set rather
than counted as a lost race — previously keyed off `S3Error::{Reqwest, Http,
Io}` and now keys off `ObjectStoreError::is_ambiguous`, which the S3 provider
maps from exactly those three variants. Everything else stays a classified
observation that fails the probe closed.
Behavior is preserved, including the paths that historically surfaced a
backend 404 as a generic storage failure rather than a media `NotFound`.
Neither `buzz-media` nor `buzz-relay` depends on `rust-s3` any more.
Signed-off-by: mozarthq <nate@mozarthq.com>
Cloud Storage's S3-interoperability endpoint accepts `If-Match` on PUT and then ignores it. Under the Git conformance probe that is not a degraded result but a semantic compare-and-swap violation: two racers predicating a write on the same revision both reported a commit, so the last writer silently destroyed the other's pointer update. Nothing above the provider can recover a lost update it was told did not happen, so Cloud Storage needs a provider that uses the native API, where the precondition is enforced. Add `providers::gcs`, implementing the seam against the official `google-cloud-storage` client with Application Default Credentials. No key file, HMAC pair, or S3 interoperability path is involved. Compare-and-swap uses native object generations rather than ETags: `ifGenerationMatch=0` is the create-only precondition and `ifGenerationMatch=<g>` the revision-matched replace. A stale precondition returns HTTP 412 and is reported as an ordinary `ConditionalWrite::Conflict`, and a pointer read takes its body and its generation from the same response. Three details are worth calling out: - **Bucket contract at construction.** The S3 provider detects versioning by writing a probe object and looking for a version id. That heuristic is meaningless on Cloud Storage, where every object carries a generation whether or not old ones are retained, so this provider reads bucket metadata instead and fails closed unless object versioning is off and soft-delete retention is zero. Either setting would leave a restorable copy behind after a delete, so a deletion could otherwise report success while the bytes stayed reachable. Checking at construction also catches configuration drift on every boot. - **Retries are owned here.** The client's own retry loop is disabled and this module runs one bounded policy: capped exponential backoff with equal jitter, honouring `Retry-After`, drawing on a throttle budget separate from the transient-error budget so pacing cannot consume the allowance for real failures. An exhausted budget surfaces as `Throttled`, reaching the caller as backpressure rather than as a lost race. A conditional write is only ever retried carrying its exact original precondition. `put_file` is the one exception, delegating to the client's resumable upload retry so a large media blob resumes instead of restarting. - **Ambiguity is resolved by rereading, not guessing.** A 412 arriving after an attempt that never got a classified answer could be another writer's commit or this writer's own. The object is reread and its body decides. `ProviderSelection`/`ObjectStoreConfig` and an async `connect()` let a deployment select its provider once, in configuration. `BUZZ_OBJECT_STORE_PROVIDER` chooses it in the relay, defaulting to `s3` so an existing deployment is unaffected. Selecting `gcs` requires `BUZZ_OBJECT_STORE_BUCKET` and deliberately does not fall back to `BUZZ_S3_BUCKET`: a Cloud Storage deployment carries no `BUZZ_S3_*` values at all, and a default bucket name would address storage nobody configured. Startup connects through the seam, so a provider's admission checks run before the relay serves traffic — a bucket whose configuration would break deletion fails the boot rather than the first delete. Signed-off-by: mozarthq <nate@mozarthq.com>
The conformance probe's S3 profile races 32 writers against one object name, three rounds over, as fast as it can. Cloud Storage publishes a roughly one-write-per-second ceiling per object name, so that shape would have spent the whole probe being throttled and then read the resulting throttles as conformance evidence — a gate that passes because the backend refused to answer is not a gate. Add a second profile the provider's own characteristics justify: a narrow race of 3 writers, 2 rounds, with same-key rounds spaced past the published ceiling (>1 s). It proves the same properties the S3 profile does, on a backend that must be paced to prove anything: - body and generation read from one response and checked against what was just committed; - a compare-and-swap on the observed generation, which must report a different one; - a replay of the superseded generation, which must conflict; - a narrow race on one generation, repeated; - the winning generation predicating the next successful write. Three rules keep pacing from becoming leniency. Two committed racers is always fatal. A commit reported with no generation is fatal wherever it appears — the caller would have nothing to predicate its next write on, and dropping the precondition is a blind overwrite. And a round that proves nothing, because every racer was throttled or too few were classified to witness a race, is re-run within a bounded budget rather than scored; exhausting the budget fails the probe. `Throttled` is never counted as a lost race. Conflicts with no acknowledged winner stay fatal, since that is a lost update announcing itself. Startup previously built the probe config from hardcoded S3 defaults, so a Cloud Storage deployment would have run the wrong gate. The defaults now come from the provider the process actually connected to, and the two environment overrides continue to apply on top for either profile. The gate stays enabled and fatal; only the shape of the evidence changed. The report and the admission log line carry the profile, the throttle and re-run counts, the shortest same-key interval actually observed, and any probe object cleanup could not remove — so a backend that is degrading is visible while it is still passing. Probe objects are deleted on the success and failure paths alike: a failed probe is the one that gets re-run. Tests drive a scripted store through the seam, because a live bucket cannot be asked to commit two writers on one generation — covering two winners, an all-throttled round re-run and then bounded, a mixed conflict/throttle round, a commit with no generation on both the create and the race path, a store that honours a stale generation, a round where everyone loses, and the measured spacing. Signed-off-by: mozarthq <nate@mozarthq.com>
…profiles Unit tests prove how the code judges answers; only a real bucket proves which answers Cloud Storage actually gives. Add the suites that close that gap, all `#[ignore]`d and additionally gated on `BUZZ_GCS_LIVE=1` so a bare `--ignored` run without credentials skips rather than fails. Credentials come from Application Default Credentials, exactly as in production, and every test works under its own uuid-scoped namespace and asserts its own cleanup emptied it. - `tests/gcs_live.rs` — provider conformance against a live bucket: bucket admission accepted and refused, permission denied refusing to hand out a client, foreign revisions never reaching the backend, create-only idempotence, compare-and-swap admitting exactly one writer, reads agreeing across full/range/stream, large objects streaming through a resumable upload, prefix pagination, idempotent deletion with per-key outcomes, and rapid same-key transitions being paced rather than failed. - `tests/gcs_scale.rs` — store-level scale evidence, additionally gated on `BUZZ_GCS_SCALE=1`: a repeated hot-pointer race that must never admit two winners, concurrent creates across distinct names that must all commit, and a sustained mixed read/write/list load reporting its latency percentiles and outcome mix. Each reports its numbers so a regression in throttling or contention behaviour is legible rather than merely non-fatal. - `api::git::store::gcs_live` — the deployment gate itself, run against the provider it was written for: the profile admitting a real bucket, the pointer cycle publishing exactly one state, content-addressed objects round tripping with corruption detected, and a chunked seed committing every sequential transition. Also documents the two conformance profiles and the provider-neutral implementation vocabulary in `docs/git-on-object-storage.md`, so the reason the Cloud Storage profile is shaped differently is recorded next to the design it belongs to rather than only in the probe source. Signed-off-by: mozarthq <nate@mozarthq.com>
Keep media, Git, and deletion domain behavior behind ObjectStore while relay and standalone process roots select GCS or S3. Add provider-neutral exact-version deletion, fail-closed provider tokens, and an explicit GCS-to-S3 cutover proof. Signed-off-by: mozarthq <nate@mozarthq.com>
Signed-off-by: mozarthq <nate@mozarthq.com>
NateJowett
force-pushed
the
mozart/gcs-object-store
branch
from
August 31, 2026 17:34
465b4ce to
c91702d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Buzz Git publication requires an exact compare-and-swap boundary. Google Cloud Storage S3 interoperability accepts
If-Matchon PUT but does not provide the native generation semantics this protocol needs, so this change adds a native GCS adapter instead of routing GCS through S3 compatibility.Provider-neutral architecture
buzz-object-store::ObjectStoreis the only storage interface consumed by media, Git, and deletion behavior. Provider construction is confined to process composition roots:BUZZ_OBJECT_STORE_PROVIDER=gcsselects native GCS and requiresBUZZ_OBJECT_STORE_BUCKET.BUZZ_OBJECT_STORE_PROVIDER=s3selects the existing S3 adapter and readsBUZZ_S3_*settings.Domain code no longer constructs S3 clients or imports provider types.
MediaStorage::newandGitStore::from_s3_configare removed; the provider-neutralwith_store/new(Arc<dyn ObjectStore>)facades remain.Safety semantics
The seam includes buffered and streaming transfer, range reads, listing, bulk deletion, immutable writes, exact-revision conditional writes, and provider-neutral exact-version enumeration/deletion.
ifGenerationMatch=0for create-only writes, and uses the observed generation for CAS and exact deletion.Portability and cutover
GCS is a deployment choice, not a key-layout or schema choice. The S3 adapter remains first-class for a future AWS deployment.
docs/git-on-object-storage.mdrecords the cutover proof:Bucket names are configuration and never appear in object keys or domain records. No AWS resources or production object copy are part of this PR.
Evidence
Local deterministic gates on the refreshed upstream base:
buzz-object-store: 47/47 unit tests passed.buzz-media: 113/113 unit tests passed.buzz-deletion: 17 passed, 9 integration tests ignored.-D warningspassed for object-store, media, deletion, relay, and test-client, all targets.Live native-GCS proof against a disposable, isolated test bucket: 12/12 passed. This covers bucket admission/refusal, exact-generation listing and deletion, foreign-token refusal, idempotent create, CAS with exactly one winner, full/range/stream agreement, prefix pagination, idempotent deletion outcomes, a 96 MiB resumable upload/stream round trip, and rapid same-key transitions paced rather than failed.
The broader relay suite reached 1001 passing and 91 ignored; 6 database-backed media tests could not acquire the absent local Postgres pool (
Sqlx(PoolTimedOut)). The storage-specific and affected suites above are green.Operational constraint
GCS has a per-object-name write ceiling, so one repository manifest pointer cannot sustain arbitrary publication frequency. Distinct repositories use distinct pointer names and remain independently concurrent. The provider-specific startup profile validates this behavior before the relay serves traffic.