Disussion: Query-time delta compensation for real-time reads of stream tables

[Teletele-Lin]

What do you think of query-time delta compensation as a direction for real-time reads of stream tables?

The idea is to combine a stream table’s materialized contents with the changes not yet incorporated by background refresh, computing the compensation at query time. Conceptually, the result would be equivalent to evaluating the defining query under the query’s MVCC snapshot, recursively accounting for upstream stream-table definitions. “Real-time” here means freshness within that snapshot.

This would be computation only: no refresh, no writes to stream-table storage, and no advancement of refresh frontiers. Background maintenance would continue asynchronously, while queries would compute the remaining delta for their snapshot. The appeal is that freshness at read time would no longer depend on how recently the materialized state was refreshed.

One question is how this would compose through a DAG. For example:

base table → stream table A → stream table B

When querying B, there may be both changes already materialized in A but not yet consumed by B, and changes in the base table that A has not incorporated yet. Computing B’s result would require composing those persisted and query-time deltas relative to B’s consumed frontier, then propagating the combined change through B’s operators without omissions or double counting. Branching and converging paths make this more involved.

From this perspective, each upstream stream table would contribute a query-local relation consisting of its stored state plus the remaining delta. A downstream computation would use that compensated relation together with the delta relative to its own consumed frontier. The interesting question is how to reason about these states and deltas consistently across the whole graph.

Does this seem like a worthwhile direction? We would be interested in thoughts on the value and trade-offs of combining asynchronous maintenance with snapshot-consistent reads, and on a clear conceptual model for composing the deltas across a DAG.