Expose stream payloads and deployment pipeline

[pragmaticAweds]

Summary

• expose complete stream fields in GraphQL, including JSON payload objects
• add stream snapshot/readable timestamp migration metadata
• include the staging/production deploy pipeline and documentation already on this branch

Verification

• bun run lint
• bun run type-check
• bun run test

Summary by CodeRabbit

• New Features

  • Added a richer GraphQL/API stream view with readable timestamps, snapshot status, remaining amount, and raw/readable payload fields.
  • Introduced support for displaying readable payloads across flow and lockup events.
  • Added CI and deployment workflows for automated checks and branch-based releases.

• Bug Fixes

  • Stream state now stays in sync after deposits, withdrawals, adjustments, and lifecycle events.
  • Improved handling of runtime environment files and startup configuration.

[coderabbitai]

Warning

Review limit reached

@pragmaticAweds, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 6 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: efa21e51-eefd-4e18-93bb-9dbcdc3356b8

📥 Commits

Reviewing files that changed from the base of the PR and between 76f87e2 and 5ccedc1.

📒 Files selected for processing (1)

• streams/src/runtime/tests/index.test.ts

📝 Walkthrough

Walkthrough

This PR introduces CI/CD workflows and deployment infrastructure, a runtime .env loader, a continuous streams indexer runtime loop with start-ledger/cursor resolution, and a canonical stream snapshot model with raw/readable event payloads persisted via migrations, storage helpers, event handlers, and exposed through the GraphQL API.

Changes

CI/CD and Deployment Infrastructure

Layer / File(s)	Summary
CI workflow and design docs .github/workflows/ci.yml, README.md, docs/stream-indexer.md, docs/plans/2026-06-28-ci-cd-deployment*.md	Adds a GitHub Actions CI workflow running lint/type-check/test on PRs and pushes, plus README/docs CI sections and CI/CD design/plan documents.
Deploy workflows, Dockerfiles, compose, and deploy script .github/workflows/deploy*.yml, api/Dockerfile, streams/Dockerfile, api/package.json, streams/package.json, .dockerignore, .gitignore, .env.example, deploy/*	Adds staging/production SSH-based deploy workflows, API/streams Dockerfiles and start scripts, docker-compose service definitions, deploy/deploy.sh, ignore-file updates, and example environment values.

Runtime Environment File Loading

Layer / File(s)	Summary
loadRuntimeEnv loader and entry-point wiring common/src/config/runtimeEnv.ts, common/src/config/tests/runtimeEnv.test.ts, common/src/index.ts, api/src/main.ts, streams/src/main.ts	Adds a .env-file-based environment loader, exports it, wires it into API/streams startup, and adds SIGINT/SIGTERM shutdown handling for the streams service.

Streams Indexer Runtime Loop and Event Fetching

Layer / File(s)	Summary
Runtime run()/close() loop streams/src/runtime/index.ts, streams/src/runtime/types.ts, streams/src/runtime/tests/*	Adds an abortable polling run() loop, idempotent close(), and a waitForPollInterval helper; run() now returns void.
Start ledger resolution and cursor flooring streams/src/app/index.ts, streams/src/app/tests/*	Adds resolveStartLedger/ensureStreamCursorFloor helpers that derive and floor the indexing start point using the Soroban client and cursor repository.
getLatestLedger, topic filters, and logging streams/src/indexer/*	Adds getLatestLedger to the Soroban RPC client, catalog-driven topic filter batching, and structured event/fetch logging.

Canonical Stream Snapshot and Readable Payloads

Layer / File(s)	Summary
Schema, migrations, and snapshot metadata common/src/db/schema.ts, common/src/db/migrations/*, common/src/db/tests/schema.test.ts	Adds payload_readable columns and stream snapshot fields (type, times, status, remaining amount, payload raw/readable) via schema and SQL migrations, plus updated snapshot metadata.
Readable payload serialization streams/src/events/readablePayload.ts	Adds serializers and per-event readable-payload factories for flow and lockup event data.
Stream snapshot storage and helpers streams/src/streamRecords/*	Expands StreamEntity/StreamInput/StreamRow/StreamStorage, adds timestamp formatting and snapshot patch/update helpers, and expands Postgres stream storage logic.
Event handlers persisting/updating snapshots streams/src/handlers/*, streams/src/adjustments/*, streams/src/flowEvents/*, streams/src/lockupEvents/*, streams/src/deposits/*, streams/src/withdrawals/*	Updates flow/lockup creation, adjustment, lifecycle, deposit, and withdrawal handlers to persist payloadReadable and update stream snapshot state.
GraphQL API exposure api/src/resolvers/*, api/src/schema.ts, api/src/server.ts, api/src/types.ts, api/src/tests/*	Adds a JSON scalar and extends schema/resolvers/types to select and expose the new snapshot and readable payload fields.
Documentation README.md, docs/stream-indexer.md, docs/plans/2026-07-03-streams-canonical-state.md	Documents the canonical streams snapshot model and continuous indexing loop.

Estimated code review effort: 4 (Complex) | ~75 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EventSource as SorobanEventSource
  participant Handler as EventHandler
  participant ReadablePayload as readablePayload.ts
  participant EventStorage as EventTableStorage
  participant StreamStorage as StreamStorage
  participant API as GraphQL API

  EventSource->>Handler: dispatch StreamIndexerEvent
  Handler->>ReadablePayload: createReadablePayload(event)
  ReadablePayload-->>Handler: payloadReadable
  Handler->>EventStorage: insert(record with payload/payloadReadable)
  Handler->>StreamStorage: updateStreamSnapshot(id, patch)
  StreamStorage-->>Handler: updated StreamEntity
  API->>EventStorage: select payload, payload_readable
  API->>StreamStorage: select stream snapshot columns
  API-->>API: map rows to GraphQL fields (payloadReadable, snapshot fields)

Loading

sequenceDiagram
  participant Main as streams/main.ts
  participant Runtime as StreamIndexerRuntime
  participant App as StreamIndexerApp
  participant Cursor as CursorRepository
  participant Soroban as SorobanRpcEventClient

  Main->>Runtime: run()
  loop until closed
    Runtime->>App: runOnce()
    App->>App: resolveStartLedger()
    App->>Soroban: getLatestLedger()
    Soroban-->>App: latestLedger
    App->>Cursor: ensureStreamCursorFloor(startLedger)
    App->>App: runStreamIndexerBatch(startLedger)
    Runtime->>Runtime: waitForPollInterval(pollIntervalMs)
  end
  Main->>Runtime: close() on SIGINT/SIGTERM
  Runtime-->>Main: run() resolves

Loading

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}