Deployment topology, scaling constraints, historical load measurements and capacity-planning scenarios. The million-user section is a planning model, not verified production capacity. For image builds and the release checklist see deployment.md; for Kubernetes see deploy/k8s/README.md.
The file service uses its configured encrypted file storage for both temporary imports and retained page attachments. Multiple file-service replicas need a shared durable store/volume and a routing design; backend statelessness does not make local filesystem data available across hosts. AI workers also retain process-local in-flight calls while durable ownership/checkpoints live in the database.
- Stateless services. Sessions, AI proposals, reminders and status history live in Postgres. Any instance of a service can answer any request, so instances can be added, removed or moved between machines freely.
- Separate services, one image.
api,ai,statusandnotifierrun from the same backend image with different commands. Each scales on its own, and a slow AI provider cannot starve the planner API. - Coordination through the database. Delivery claims use row/advisory locks to prevent concurrent owners; status probing also coordinates its work. External SMTP acceptance can remain ambiguous after a crash. Booking mail records uncertain delivery instead of blindly retrying, without promising exactly-once delivery.
- Pooling everywhere. Instances keep small pools (
DB_POOL_MAX); PgBouncer multiplexes thousands of client connections onto a few dozen Postgres connections.
HTTP services have explicit composition roots while sharing their image, database, HTTP shell and coordinated migrations. This permits separate runtime scaling; it does not establish independent release or data ownership.
The planner uses bounded best-effort deltas between primary repeatable-read snapshots, with a five-minute reconciliation interval while active/connected. Known access changes force reconciliation and prune revoked teams. Failed requests retain valid state and retry; offline intervals have no freshness guarantee. Bootstrap still collects the account's full visible item set, so SQL paging alone does not bound total response size or client memory.
Synthetic local app.inject samples for GET /items/reconcile were 120,738 bytes
for 100 items, 1,208,840 bytes for 1,000 and 6,052,840 bytes for 5,000. A repeated
5,000-item sample varied from about 232 ms to 1.02 s in the shared test process.
These establish fixture baselines, not production/native capacity, peak memory,
percentiles or an agreed latency target.
Assistant admission retains ceilings of 8 total, 4 interactive, 2 background and 2 overnight. Compatible claimers check one persisted policy revision under the slot lock. Policy decreases drain existing leases; they do not kill work. Old claimers must be stopped or fenced at cutover. Budgets remain separate from slots. Fairness measurement is by team when scope is known, with private owners and unscoped jobs reported separately. No scheduler change, starvation bound or capacity increase follows from synthetic queue-age attribution fixtures.
flowchart LR
Clients[Web, mobile, desktop] --> LB[Load balancer / ingress<br/>TLS, health checks]
LB --> G1[Gateway 1]
LB --> G2[Gateway N]
G1 & G2 --> API[api x N]
G1 & G2 --> AI[ai x N]
G1 & G2 --> ST[status x 2]
API & AI & ST & NT[notifier x N] --> PB[PgBouncer]
PB -->|writes, fresh reads| P[(Postgres primary)]
PB -->|"<db>_read"| R[(Read replicas)]
P -. streaming replication .-> R
| Layer | Scales by | Notes |
|---|---|---|
| Load balancer | Managed (ALB, Cloud Load Balancing, ingress) | Terminates TLS, health-checks gateways on /health |
| Gateway (nginx) | More instances behind the load balancer | Routes /ai/*, /status, everything else; keep-alive upstreams; least_conn |
api |
Horizontal, CPU-based autoscaling | Planner, auth, teams, admin |
ai |
Horizontal | Mostly waiting on providers; scale on concurrency |
realtime |
Horizontal, on CPU and memory | Holds the apps' open streams; one LISTEN connection per copy fans news out |
status |
2 for availability | One prober at a time (advisory lock); report cached 15 s |
notifier |
Horizontal | Parallel lanes (NOTIFIER_CONCURRENCY) per instance |
files |
1, on a volume; shard by import id if needed | Encrypted import staging and retained page attachments; shared storage required |
converter |
Horizontal | Claims files and pages with SKIP LOCKED; runs Tesseract itself (TESSERACT_WORKERS lanes) |
formula |
1, ~1–2 GB (optional) | pix2tex for equations on scans; only lines that look like maths are sent |
ocr |
More containers (--scale ocr=N), ~12 GB each |
CPU only, one page at a time per container; set OCR_WORKERS to match |
| PgBouncer | 2+ instances, scaled by hand | Transaction pooling; <db>_read route to a replica |
| Postgres | Vertical primary, horizontal read replicas | Managed service recommended in production |
Live news (GET /events) and live documents (GET /events/docs/:id) keep a connection open for
as long as an app is. On the API those would pin connections to copies that otherwise serve short
requests and scale on CPU; a deploy or scale-down would drop them mid-request. The realtime copies
hold nothing but streams: each keeps one Postgres LISTEN connection and forwards news to the
streams connected to it. The API (and the worker) raise news with NOTIFY inside the transaction
that made the change, so any API copy reaches readers on any realtime copy, and nothing is sent
for a change that rolled back. Each stream closes itself after 15 minutes and the apps reconnect
with backoff, which spreads readers over new copies and picks up teams joined since.
Outside agents that follow Orbyn (MCP subscriptions/listen) are streams too, so they are held
here and never by the stateless mcp copies: the gateway sends a POST /mcp whose Mcp-Method
header is subscriptions/listen to realtime (map $http_mcp_method; a canary Ingress by header in
Kubernetes). Each listen stream hears the same LISTEN news, filtered to what its connection
can read, closes within 15 minutes or at its credential's end, and closes at once when the
connection's access changes (orbyn_auth). An agent's long jobs (MCP tasks) are rows in
mcp_tasks, so any mcp copy answers tasks/get.
Presence check-ins (POST /presence/heartbeat, about one a minute per open app) are ordinary
short writes on the API, one row per device. Changes made offline are replayed with an
Idempotency-Key; the first answer is kept in idempotency_keys for a day and a repeat gets it
back, so a retry after a dropped connection never does anything twice. Old keys are pruned as
requests come in.
Every piece is addressed by configuration, not by Docker's service names.
Gateway. gateway/nginx.conf.template is rendered at start by
gateway/entrypoint.sh. Point it at instances anywhere:
GATEWAY_API_SERVERS="10.0.1.10:8000 10.0.1.11:8000 api.internal:8000"
GATEWAY_AI_SERVERS="10.0.2.10:8000"
GATEWAY_STATUS_SERVERS="10.0.3.10:8000"
GATEWAY_RESOLVER=10.0.0.2 # your VPC DNS, for hostnames
GATEWAY_TRUSTED_PROXIES="10.0.0.0/16"Hostnames are re-resolved every 10 seconds, so a DNS name that returns several addresses (Consul, Cloud Map, a Kubernetes headless Service) load balances without restarting the gateway. Unreachable instances are skipped after 3 failures for 10 seconds.
Services. Each host runs the backend image with the command for its service and these settings:
| Setting | Value on a multi-host deployment |
|---|---|
DATABASE_URL |
PgBouncer (or the managed pooler) for the primary |
DATABASE_READ_URL |
PgBouncer's <db>_read route or a replica endpoint |
TRUST_PROXY |
true behind the gateway |
RATE_LIMIT_PER_MINUTE |
A generous per-instance backstop, or 0 to leave it to the gateway |
STATUS_GATEWAY_URL, STATUS_API_URL, STATUS_AI_URL, STATUS_REALTIME_URL |
Internal URLs the status service probes |
SECRETS_KEY |
The same value on every ai and api instance |
Migrations run once per release from one place (node backend/dist/migrate.js
against the primary directly) before new instances start.
Health checks. Every HTTP service has two endpoints:
GET /live: liveness. No database, so a brief database outage never makes an orchestrator restart every instance at once.GET /health: readiness. Includes a database check; a failing instance is taken out of rotation until it recovers.
Put a managed load balancer or ingress in front of two or more gateways (or,
on Kubernetes, use the ingress instead of the gateway; see deploy/k8s).
- Health-check gateways on
/health(the gateway answers without a backend hop). - Set
GATEWAY_TRUSTED_PROXIESto the load balancer's address ranges. The gateway then takes the client address fromX-Forwarded-For, so rate limits and logs apply per end user rather than to the load balancer. - In production leave
GATEWAY_RATE_LIMIT_EXEMPTempty; its default only exists so a load test from the same machine is not throttled. - Rate limits are per gateway instance: with N gateways a client can reach up to N times the configured rate. For strict global limits, enforce them at the load balancer or WAF, or add a shared store (see "Next steps").
PgBouncer runs in transaction mode (pgbouncer/pgbouncer.base.ini:
max_client_conn 5000, default_pool_size 40). Keep
service instances x DB_POOL_MAX <= PgBouncer max_client_conn x PgBouncer instances
PgBouncer instances x (default_pool_size + reserve_pool_size) < Postgres max_connections
Managed poolers (RDS Proxy, Cloud SQL connector pooling, Azure Flexible
Server's PgBouncer) replace this container. Set SERVER_TLS_SSLMODE=require
when PgBouncer talks to a database that expects TLS.
Writes, transactions and anything security-sensitive (sign-in, token checks,
permission checks inside writes) always use the primary. Reads that tolerate
a moment of replication lag go to DATABASE_READ_URL when it is set:
- item lists and task detail, notifications, the team list;
- the AI assistant's planner lookups;
- the admin console (users, teams, audit log, stats);
- the public status report.
Read-your-writes. For 5 seconds after a client writes, the shared API
client sends X-Orbyn-Consistency: primary on its reads, and the server
serves those from the primary. A user always sees their own change even if a
replica is a moment behind. Without DATABASE_READ_URL everything uses the
primary and the header has no effect.
Locally, the replica Compose profile runs a streaming replica:
# First start of a new database volume only: the primary allows replicas.
PGBOUNCER_READ_HOST=postgres-replica \
DOCKER_DATABASE_READ_URL=postgres://orbyn:PASSWORD@pgbouncer:6432/orbyn_read \
docker compose --profile replica up -dThe primary's first-start script (deploy/postgres/primary-replication.sh)
allows replication connections; for a database that already exists, add the
same line to pg_hba.conf and reload. The replica
(deploy/postgres/replica-entrypoint.sh) clones the primary with
pg_basebackup and runs as a hot standby.
In production use a managed Postgres with read replicas (RDS/Aurora, Cloud SQL, Azure Flexible Server, Crunchy Bridge). They provide automated failover, point-in-time recovery and replica endpoints; set that endpoint as the read route. Failover of the primary is the provider's job; services reconnect through the pooler.
- Hot paths are indexed: open work by due date (
items_due, partial onstatus <> 'done'), per-owner and per-team lists, notification claims. - The scheduler only scans a bounded window (due within the next 7 days, or in the last day).
- Expired AI proposals are removed after a day, and finished reminder records after 90 days.
- Beyond tens of millions of items, partition
items,item_updatesandnotificationsby time or by owner hash; the code addresses rows by id and owner, which suits either scheme.
The apps refresh every 30 seconds; at scale, that polling is most of the traffic.
- Conditional GETs. Every successful GET carries a weak
ETag. The shared client remembers the last body per signed-in path and sendsIf-None-Match. Unchanged data comes back as a bodyless304, which the client turns into the cached result. - Status page. The gateway micro-caches
GET /status, and the status service caches its report for 15 seconds, so the public page costs almost nothing however many people watch it. - Compression and keep-alive at the gateway, with persistent upstream connections to every instance.
A load test on one MacBook: the load generator, three api processes, one
ai, one status, the gateway, PgBouncer, a primary and a streaming replica,
all sharing the same CPU. The gateway reached the processes through
host networking, as it would reach separate machines. Numbers are therefore a
floor for a single laptop, not per-server capacity.
LOAD_URL=http://127.0.0.1:18008 LOAD_SECONDS=15 LOAD_CONCURRENCY=64 npm run load -w backend| Scenario | Requests/s | p50 | p95 | p99 | Errors |
|---|---|---|---|---|---|
Gateway /health |
8,560 | 6.9 ms | 11.0 ms | 15.1 ms | 0% |
GET /status (cached) |
3,642 | 16.8 ms | 24.3 ms | 34.1 ms | 0% |
GET /me |
2,951 | 17.4 ms | 39.5 ms | 87.0 ms | 0% |
GET /items (50 items, replica) |
1,143 | 51.0 ms | 90.1 ms | 159.4 ms | 0% |
POST /items (primary) |
1,531 | 37.6 ms | 58.1 ms | 89.0 ms | 0% |
CPU time was spread almost evenly across the three api processes (about
18 seconds each), confirming the gateway balances across instances.
Replication was checked on the same stack: PgBouncer's primary route is not in
recovery, its _read route is; a write on the primary was visible on the
replica after 4 ms; the replica rejects writes.
Assumptions, to be replaced with real telemetry:
| Quantity | Assumption | Result |
|---|---|---|
| Registered users | 1,000,000 | |
| Peak concurrently open apps | 5% | 50,000 |
| Background refreshes | 1 per 30 s per open app | ~1,700 req/s |
| Interactive requests | ~3 per minute per open app | ~2,500 req/s |
| Peak total | with 2x headroom | ~8,500 req/s |
Sizing from that peak:
api: at a conservative 300–400 list or write requests per second per vCPU, 20–30 one-vCPU instances cover the peak; the Kubernetes autoscaler allows 3–50.- Database: most refreshes are unchanged and end as
304s, but they still run their query. A primary with 8–16 vCPUs handles the writes; 2–3 read replicas carry the lists. Pooling keeps Postgres at a few hundred connections regardless of instance count. ai: bound by provider latency and per-user limits (10 chats per minute), not CPU; scale on in-flight requests.notifier: one instance with 8 lanes sends thousands of reminders a minute; add instances if the queue grows.- Gateways: 2–3 behind the load balancer for availability; one nginx instance alone serves the peak.
These are not built yet; add them when telemetry says so.
- Push instead of polling. Server-sent events or WebSockets for item changes would remove most refresh traffic; the 30-second poll becomes a fallback.
- Shared rate limits. A Redis-backed limiter (for example
@fastify/rate-limitwith a Redis store) gives exact global limits across gateways and instances. - Server-side cache. Redis for hot reads such as
GET /meand team membership, invalidated on write. - Observability. Prometheus metrics and OpenTelemetry traces per service, with alerts on p95 latency, error rate, queue depth and replication lag.
- Multi-region. Read replicas per region with the primary in one; route users to the nearest region for reads.