feat(offline-sync): production-grade offline sync & idempotent queue (#93) - #2
Open
dorisadams wants to merge 15 commits into
Open
feat(offline-sync): production-grade offline sync & idempotent queue (#93)#2dorisadams wants to merge 15 commits into
dorisadams wants to merge 15 commits into
Conversation
…ssion flow Add a realistic backend/SDK integration test suite for Soroban RPC against mock Stellar nodes, frontend contract submission/error-handling tests, and mobile offline-queue contract retry tests, all anchored on a shared fixture (fixtures/contract-flow.json) so the three layers assert against one consistent ledger/transaction scenario. Wire the suite into CI via a new scripts/run-regression-harness.sh and a regression-harness job in ci-quality.yml, and document the strategy in docs/integration-testing.md and docs/regression-harness.md. Closes Stellar-Insightss#99
…9-multi-layer-regression-stellar-rpc test: multi-layer regression suite for Stellar RPC and contract submission flow
- Add backend /debug/* endpoints (ping, env, flags) gated by guard_dev_only middleware; blocked with HTTP 403 in production and staging - Add frontend debug helpers (getNetworkState, getWebSocketHealth, getCacheStatus, getDebugReport, installDebugConsoleHelpers) that return null when NODE_ENV !== development - Add mobile debug service and useDebugDiagnostics hook covering offline queue state, sync status, and notification channel health; all gated on __DEV__ - Add docs/debugging-guide.md covering end-to-end local setup, WebSocket debugging, common failure scenarios, and env variable reference - Wire debug routes into the backend router under /debug with dev-only guard - 35 tests across frontend and mobile verify production safety and correct dev-mode behavior; no secrets exposed in any diagnostic output Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…#127) Flips `unwrap_used`, `expect_used`, `panic` from `allow` to `warn` in `backend/Cargo.toml` and tightens `.github/workflows/clippy.yml` + `ci-quality.yml` so `cargo clippy --lib` denies them. The three non-test `.expect()` calls in `backend/src/rpc/stellar.rs` are replaced with safe `match`+`warn` fallbacks / typed errors. No scaffolding for Stellar-Insightss#93 or Stellar-Insightss#104 in this PR — those PRs register their own modules. This keeps the lint fix scope minimal and conflict-free. Refs Stellar-Insightss#127
Stellar-Insightss#93) Replaces the stub `mobile/src/services/database.ts` with a real SQLite (react-native-sqlite-storage) implementation — schema for corridors/anchors/assets/sync_queue, schema-version migrations, idempotent `enqueueSync()` keyed on `dedup_key`, exponential backoff in `markSyncFailed()`. `mobile/src/hooks/useOfflineCaching.ts` mirrors cache writes into SQLite on a best-effort, single-entry basis (the mirror fires BEFORE `setCache` to close the MMKV-fresh / SQLite-stale race). Adds `useStaleLocalStorage` to `frontend/src/hooks/useLocalStorage.ts` for stale indicator / invalidate / metadata UI. Backend gets a new idempotent `QueueProcessor` (in `backend/src/queue/{mod,types,replay}.rs`) with dedup_key short-circuit, retry-then-give-up, plus 5 unit tests covering idempotency, retries, and max-retry give-up. Refs Stellar-Insightss#93
Enable clippy::unwrap_used, clippy::expect_used, and clippy::panic in Cargo.toml (changed from 'allow' to 'warn') to enforce proper error handling across the codebase. Production fixes: - handlers/job_monitoring.rs: replace 20+ chained .unwrap() calls on JSON Value with ok_or_else() propagation; extract parse_job_entry() helper with a get_field! macro for typed, fallible field access - rpc/stellar.rs: remove .expect() on pagination map lookup by using .and_then(|v| v.as_object_mut()) guard instead Infallible sites (annotated with #[allow] + SAFETY comments): - rpc/stellar.rs, services/contract.rs, services/price_feed.rs, telegram/client.rs: reqwest Client::builder().timeout().build() cannot fail with only a timeout configured - webhooks/mod.rs: HmacSha256::new_from_slice() accepts any key size - shutdown.rs: OS signal handler installation — panic is correct - services/realtime_broadcaster.rs: shutdown_rx.take() programmer contract (start() must only be called once) - observability/tracing.rs: tracing-subscriber span contract - observability/metrics.rs, job_metrics.rs, rpc/metrics.rs: Prometheus lazy_static! registration — file-level #![allow(clippy::expect_used)] - logging/redaction.rs: Regex::new() on hardcoded valid patterns — file-level #![allow(clippy::unwrap_used)] Test code: - main.rs + lib.rs: add #![cfg_attr(test, allow(clippy::unwrap_used, clippy::expect_used))] at crate roots to cover all test modules without per-module annotations CI already enforces -D warnings on clippy via ci-quality.yml.
…ghtss#93) Adds a durable mobile SQLite-equivalent local persistence layer (corridors/anchors/assets/sync_queue tables), replay-on-reconnect logic for queued offline mutations, frontend cache staleness/invalidation support, and an idempotent backend replay queue, tying the three layers together under a shared id-based reconciliation contract. - mobile/src/services/database.ts: table-oriented local store (MMKV backed, SQL-swap-friendly) with CRUD helpers and a sync_queue table for durable offline mutations. - mobile/src/hooks/useOfflineCaching.ts: replayPendingSyncActions() and a reconnect effect that drains the queue and invalidates stale cache. - frontend/src/hooks/useLocalStorage.ts: additive ttlMs/isStale/ invalidate support, backward compatible with existing 3-tuple callers. - frontend/src/lib/api-client.ts: apiReconcile() wrapping the existing /api/rpc/reconcile endpoint for stale-cache recovery. - backend/src/queue/mod.rs: OfflineSyncQueue, a concurrency-safe idempotent ledger for replayed offline actions (atomic reservation, safe retry on failure). - docs/offline-sync.md: architecture, the id-based reconciliation contract, and failure modes. - scripts/verify-offline-sync.sh: runs the mobile/frontend/backend test suites for this feature. Note: backend/ currently fails to build on main due to pre-existing dependency-version drift unrelated to this change (redis/digest/hmac/ stellar-xdr API mismatches, ~140 errors across files this PR does not touch). `cargo check --lib` attributes zero errors to backend/src/queue/; its tests could not be executed in this environment as a result and should be re-run once that breakage is fixed. Closes Stellar-Insightss#93
…rage and mobile database (Stellar-Insightss#93)
…cted test to avoid panic from build_message rejection (Stellar-Insightss#93)
…ams slice)
Adds backend/src/debug/ {mod,tests}.rs with SI_DEBUG-gated helpers (DebugInspector, PerformanceTimer, debug_timer! macro, route-table logger) plus 9 gating tests. Adds docs/debugging/{INDEX, backend-workflows, websocket-troubleshooting, PR_DESCRIPTION}.md as the dorisadams slice of issue Stellar-Insightss#104. Frontend slice owned by @OsejiFabian; mobile slice owned by @Damidesign. lib.rs gains a single `pub mod debug;` line.
…1-66-debugging-diagnostics-workflows feat: add debugging diagnostics and local dev workflow docs (#51 #66)
…c-prevention-lints fix(lints): enable panic prevention lints and fix all flagged sites
…04-debug-workflows-backend Closes Stellar-Insightss#104 — 📚 Comprehensive Integration & Debugging Workflows (dorisadams slice)
…line-sync-local-persistence feat: production-grade offline sync & local persistence
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes Stellar-Insightss#93 — replaces stub database.ts with SQLite, adds idempotent backend queue with replay-on-reconnect.