Skip to content

Shielded sync: persist the Railgun DB — cold sync is ~11 min/launch (in-memory re-sync) #12

Description

@hellno

Problem

The in-app shielded-balance sync (deckard-core::shielded::ShieldedHandle, landed in #11) does a full cold sync on every launch because the Railgun provider uses kohaku's in-memory MemoryDatabase. Measured on mainnet:

RPC Cold sync wall-clock Notes
publicnode (free, the default) stalled >14 min, never completed rate-limited at Railgun getLogs/Subsquid volume
Alchemy archive 647 s (≈10.8 min) completed; CPU 1.8–6.6% the whole time

The app degrades honestly throughout (shows "Public · private balance still syncing", never a fake balance), so this is a UX/perf issue, not a correctness one.

Root cause (measured, not guessed)

A probe of the two syncer stages:

chain head:    25,280,659
subsquid head: 25,280,573      → RPC gap is only 86 blocks (1 getLogs batch)
full history span: 10,587,646 blocks

So the RpcSyncer gap-fill is trivial (~1 s). The entire ~11 min is the SubsquidSyncer paging the full mainnet Railgun history (10.5M blocks → millions of commitments + nullifiers). CPU stays low → it's latency-bound (hundreds of sequential GraphQL page round-trips), not note-decryption-bound.

Therefore "Subsquid-only" and RpcSyncer config tweaks (batch_size/batch_delay) save ~1 s and do NOT help. The Subsquid page size (20000) isn't exposed for tuning. Downloading the full UTXO tree is fundamental to Railgun's model — you can't know your balance without it.

The fix: persist the Railgun DB

kohaku supports it cleanly: RailgunBuilder::with_database(Arc<dyn Database>). The Database trait is a 3-method KV store (get/set/delete); kohaku even ships a ~40-line FilesystemDatabase (currently commented out of its mod.rs). So:

  • Implement a small disk-backed Database in deckard-core (copy the fs.rs pattern) and pass it via with_database. Turns ~11 min/launch into ~11 min once, then incremental (seconds).
  • Security requirement: the DB stores synced UTXO data = the user's private notes/balances. kohaku keeps key material out of it, but for a privacy wallet, writing the private balance to a plaintext file is a privacy-at-rest regression. Encrypt values at rest (reuse the keystore's XChaCha20-Poly1305 envelope pattern, keyed off the session). This makes it a small feature, not a 40-line drop-in.

Workaround for demos now

  • Use a fast archive RPC (Alchemy/Infura) via Settings → Custom RPC; the free publicnode default stalls.
  • Pre-sync once (~11 min) and keep the app open (in-memory DB is lost on close). When "private balance still syncing" resolves to the Private/Public split, the demo is ready.

Related deferred follow-ups (from the #11 codex review)

  • Sync timeout/backoff in shielded.rs — needs a tokio time feature; would make a bad-RPC stall fail honestly instead of "syncing forever" (directly relevant: publicnode stalls indefinitely today).
  • Zeroize the viewing key end-to-end (it's a String on the wire + in ViewOnlySigner).
  • Commit an SDK-generated fixture for the derivation KAT's provenance.

Evidence

Temporary [MEASURE] eprintln instrumentation in crates/deckard-core/src/shielded.rs (uncommitted) produced the timings above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions