walshadow consumes PostgreSQL physical WAL, replays filtered WAL in a shadow PostgreSQL process, and reconstructs committed rows for ClickHouse
Use docs for supported behavior and operating procedures, plans for unfinished work, and linked source for implementation
Physical tuples need catalog state from when they were written. A static schema snapshot cannot follow DDL or relation rewrites. Shadow replays source catalog changes using PostgreSQL itself, while walshadow keeps versioned descriptors for row decoding and uses PostgreSQL conversion for types it cannot decode locally
Filtering preserves WAL positions while replacing unwanted records or blocks with valid placeholders. This keeps compatibility work in WAL transformation instead of requiring a PostgreSQL recovery fork. By default, shadow retains catalogs and recovery state, not ordinary user-table contents. Shadow TOAST mode also retains selected physical data, see shadow TOAST storage
WalStream retains original records for decoding and rewrites user-table
records to no-ops for shadow replay. Shadow receives filtered bytes through
walshadow's sender, with local segments as archive fallback
CatalogCapture holds publication at schema boundaries, reads shadow at an
exact replay position, persists descriptors, and attaches SchemaEvent to
XactBuffer. Queued row processing uses that history when decoding and
planning committed transactions
BufferingDecoderSink and ReorderSink share one record-queue worker
ReorderSink plans each commit and places its rows onto the batcher in order
[ch].inserter_pool_size sizes the inserter pool
Each inserter owns a ClickHouse connection and can take any sealed batch
Sequence numbers identify work slices, not necessarily whole transactions Only a commit's final slice publishes its LSN, after all earlier work finishes Bounded queues and a shared payload budget limit work in flight; transaction and plan data can spill to disk
| Diagram | Scope | Implementation |
|---|---|---|
| Catalog capture and DDL | Historical tuple layouts and ordered schema effects | capture, reorder |
| TOAST and type conversion | Historical large values and PostgreSQL conversion | resolver, oracle |
| Shadow TOAST storage | PostgreSQL-backed large values and physical WAL routing | reader, filter |
| Bootstrap | Backup visibility, concurrent WAL, and initial-load publication | backup, window |
| Restart and cleanup | Durable progress, retained history, and timeline crossing | manifest, status loop |
Streaming wiring lives in stream/, queue ownership in queueing_record_sink.rs, and pool assembly in pipeline/mod.rs
SVGs are editable source. Rectangles identify components, dashed enclosures identify processes or worker groups, narrow bars identify queues, and cylinders identify stored state. Solid arrows carry data; dashed arrows carry progress or control. Labels name messages, protocols, or state transferred
Use dark colors: warm neutral backgrounds and text, blue data paths, orange control paths, green catalog paths, magenta stored state, and yellow ClickHouse borders
Keep diagrams and high-level explanations together here. Check connections against linked source before updating diagrams. Leave data structures, protocol layouts, function walkthroughs, and exhaustive terminology in source