Plan transactional outbox/retry strategy for save-publish-dispatch consistency (deferred)

[bartul]

Parent: #66

type = decision
status = approved
implementation status = deferred

Problem

Current save -> publish -> dispatch flow can leave state and integration effects out of sync on partial failure.

The current Rondel consistency boundary is Rondel.materialize, which sequences:

1. Save state.
2. Publish domain events.
3. Dispatch outbound commands to other bounded contexts.

If save succeeds and a later publish/dispatch step fails, state has already moved forward. For paid rondel movement, that can leave a pending movement in Rondel state without a durable Accounting command, or leave Accounting/Rondel reconciliation dependent on non-durable bus delivery.

Current code nuance

The terminal sandbox is intentionally in-memory and fire-and-forget:

• RondelHost.Execute and AccountingHost.Execute return enqueue acknowledgment only.
• The terminal bus suppresses subscriber exceptions, so a successful Publish call means the event was offered to current subscribers, not that every subscriber handled it successfully.
• Accounting is currently stateless and auto-approves charges.

This means terminal behavior is acceptable for the current sandbox, but it is not a durable/web consistency guarantee.

Failure matrix

Step	Failure mode	Current behavior	Risk
Save fails	Store returns Error	Handler fails before publish/dispatch	Clean failure; no later effects should occur
Save succeeds, publish delivery/handling fails	Bus subscriber throws or delivery is not durable	Terminal bus suppresses subscriber exceptions	State/event divergence; publish success does not prove handling
Save succeeds, dispatch fails	Dispatch returns Error or target is unavailable	Handler fails after state save	State saved but outbound command missing
Dispatch enqueue succeeds, Accounting processing fails	Accounting mailbox later fails	Caller already received enqueue acknowledgment; notification may be published	Rondel has pending movement, Accounting may not process charge
Accounting publishes return event, Rondel handling fails	Return event delivery/handling is non-durable	Event can be missed or swallowed in terminal bus path	Accounting has outcome, Rondel remains pending
Relay redelivers a message in future durable flow	Relay crashes after delivery before acknowledgment	Duplicate delivery is expected under at-least-once delivery	Consumers must be idempotent

Decision

Adopt transactional outbox plus idempotent inbox for durable/web deployments.

The durable commit must atomically persist domain state changes and outbound messages. A relay then delivers pending messages with at-least-once semantics. Consumers must be idempotent through an inbox/processed-message mechanism or equivalent business-level idempotency.

This is the selected strategy because:

• outbox prevents state committed, message lost;
• inbox/idempotent consumers handle duplicate delivery, which outbox relay cannot avoid;
• the model aligns with future HTTP 202 Accepted/eventual consistency semantics;
• it keeps the terminal sandbox simple while preserving a clear path to durable behavior.

Non-decision for current terminal sandbox

Do not implement an in-memory outbox for the current terminal sandbox.

An in-memory outbox would not survive process failure and would not provide the durability guarantee this decision is about. The current supervised mailbox and in-memory store remain acceptable for local sandbox work.

Implementation direction

When implementation starts, prefer an explicit effect commit boundary over transparent middleware capture.

Target shape:

• introduce RondelEffects as data for state, domain events, and outbound commands;
• introduce CommitRondelEffects as the boundary where infrastructure chooses direct commit or durable outbox commit;
• preserve current terminal direct-commit behavior while making the consistency boundary explicit and testable.

Transparent middleware capture may be used as a tactical bridge if needed, but it is not the target architecture. It gives Save, Publish, and Dispatch two meanings depending on context, which is too implicit for the permanent consistency boundary.

Follow-up work

• type = improvement: Introduce explicit Rondel effect commit boundary #129 - Introduce explicit Rondel effect commit boundary.
• type = improvement: Design durable outbox storage and relay #130 - Design durable outbox storage and relay.
• type = improvement: Add inbox and processed-message tracking for inbound delivery #131 - Add inbox and processed-message tracking for inbound delivery.
• Implement Stateful Accounting Domain with Balance Tracking #49 - Make Accounting stateful and idempotent, including duplicate charge handling.

Acceptance criteria

• Failure matrix documented.
• Strategy selected for later implementation.
• Decision explicitly records outbox plus inbox, not outbox alone.
• Terminal sandbox non-goal is documented.
• Implementation target is clarified as explicit effect commit boundary.
• Follow-up implementation issues are linked.

Consequences

Durable/web readiness requires more than replacing the bus with retries. The architecture needs a named commit boundary, durable outbox storage and relay, and idempotent consumers. Implementation remains deferred until the durable persistence/web phase, but future work should align with this decision.

[bartul]

Research Report: Transactional Outbox/Retry Strategy for Save-Publish-Dispatch Consistency

Issue: #76 — Plan transactional outbox/retry strategy for save-publish-dispatch consistency (deferred)
Parent: #66 — Architecture backlog: sandbox hardening toward web-readiness
Date: 2026-03-15

---

Problem Summary

The current save → publish → dispatch flow in the Rondel domain can leave state and integration effects out of sync on partial failure. The materialize function (Rondel.fs:400-423) executes three sequential IO steps:

1. Save state to persistence
2. Publish domain events to the bus
3. Dispatch outbound commands to other bounded contexts (e.g., Accounting)

If any step fails after a prior step succeeds, the system enters an inconsistent state. For example: state saved but the charge command to Accounting never dispatched, or the charge dispatched but the resulting InvoicePaid event never delivered back to Rondel.

This research designs a persistence-agnostic outbox/retry strategy for the web-ready phase, covering cross-BC communication and future external integrations (webhooks, message queues).

---

Architecture Impact

Affected Components

Core Domain Layer (src/Imperium/)

• Rondel.fs — materialize function (the consistency boundary)
• All four Rondel handlers: setToStartingPositions, move, onInvoicePaid, onInvoicePaymentFailed
• Accounting.fs — stateless skeleton, publishes events via deps.Publish
• RondelDependencies record: { Load; Save; Publish; Dispatch }

Infrastructure Layer (src/Imperium.Terminal/)

• Bus.fs — IBus with type-based pub/sub, invokeSafely swallows handler errors
• Store.fs — InMemoryRondelStore, ConcurrentDictionary, always returns Ok()
• RondelHost.fs — wires domain dependencies, SupervisedMailbox for sequential processing
• AccountingHost.fs — publishes AccountingEvent back to bus
• SupervisedMailbox.fs — sequential message processing with catch-all error handler
• Program.fs — composition root with lazy thunks for circular dependency resolution

Execution Path Trace (Move Command)

User → RondelHost.Execute (fire-and-forget Post)
  → SupervisedMailbox processes sequentially
    → Rondel.move handler
      → Load state from Store
      → Move.execute (pure logic) → (state, events, commands)
      → materialize:
          1. Save state to Store         ← can fail (throws)
          2. Publish RondelEvent to Bus  ← errors swallowed per-handler
          3. Dispatch to Accounting      ← can fail (throws)
             → AccountingHost.Execute (fire-and-forget Post)
               → Accounting auto-approves
               → bus.Publish AccountingEvent
                 → RondelHost receives InvoicePaid
                   → onInvoicePaid handler
                     → materialize again (save, publish, no dispatch)

Failure Matrix

Step	Failure Mode	Current Behavior	Inconsistency
Save succeeds, Publish fails	Bus handler throws	Swallowed by invokeSafely — state saved but downstream consumers miss event	State/event divergence
Save succeeds, Dispatch fails	Accounting unavailable or returns Error	failwith → SupervisedMailbox catches → SystemNotification	State saved but charge never sent — paid move recorded without billing
Save fails	Store returns Error	failwith → nothing persisted, no events published	Clean failure (no inconsistency)
Dispatch succeeds, return event fails	Bus delivery of AccountingEvent fails	RondelHost never receives InvoicePaid	Charge sent, payment confirmed in Accounting, but Rondel stuck with pending movement
Save succeeds, Dispatch succeeds, Accounting fails internally	AccountingHost mailbox error	SystemNotification published	Rondel state saved with pending movement, Accounting never processes charge
Duplicate processing	Message replayed after partial failure	Some idempotency exists (onInvoicePaid ignores unknown pending movements) but move is not idempotent	Potential duplicate state transitions

---

External Research Findings

Transactional Outbox Pattern

Events/commands are written to an "outbox table" in the same database transaction as the state change. A separate background process (polling publisher or CDC) reads the outbox and relays messages. Guarantees at-least-once delivery.

An F# implementation by Marcin Golenia demonstrates the pattern needs only 2 types and 2 functions. A Commit function stores messages atomically with state; an Execute function polls and publishes pending messages.

Inbox / Idempotent Consumer Pattern

Each consumer tracks processed message IDs. Before processing, check if the ID exists — if so, skip. Processing + ID recording happen atomically. Achieves effective exactly-once semantics when combined with at-least-once delivery from the outbox.

Milan Jovanovic's analysis shows the pattern uses a MessageConsumer table with composite key (MessageId, ConsumerName). For external services, idempotency keys are passed through.

Saga / Process Manager

Oskar Dudycz's comparison distinguishes:

• Saga (stateless): Routes events to commands based on simple rules. More flexible, less overhead.
• Process Manager (stateful): State machine making decisions based on current state + incoming events.

The current Rondel move → charge → invoice flow is already an implicit choreography-based saga. The PendingMovement state acts as saga state. Superseding moves with void charges are compensating transactions.

Key Insight

Industry consensus: "exactly-once" = at-least-once delivery (outbox) + idempotent processing (inbox). These are complementary, not competing patterns.

Sources

• Transactional Outbox — microservices.io
• Outbox pattern in F# — Marcin Golenia
• The Outbox Pattern — Kamil Grzybek
• Outbox/Inbox patterns and delivery guarantees — Oskar Dudycz
• Saga and Process Manager — Event-Driven.io
• Idempotent Consumer Pattern — Milan Jovanovic
• Idempotent Consumer — microservices.io
• Functional Event Sourcing Decider — thinkbeforecoding
• Implementing the Outbox Pattern — Milan Jovanovic

---

Hypothesis Evolution

Hypothesis	Initial Status	Final Status	Reasoning
H1: Transactional Outbox	Active	Active (subsumed by approaches A, C)	Core mechanism validated; question became how to integrate it
H2: Formalized Saga/Process Manager	Active	Weakened → supplementary	Real problem is delivery reliability, not coordination logic. Existing implicit choreography works.
H3: Idempotent retry alone	Active	Reframed as complementary	Doesn't solve "state saved but event never published". Needs outbox underneath.
H4: Outbox + Inbox combined	New (Phase 3)	Active, strong	Industry consensus for exactly-once semantics
H5: Lightweight functional outbox	New (Phase 3)	Eliminated	Doesn't handle process crashes between save and delivery

---

Decision Criteria

Ranked by importance for this specific problem:

1. Architecture consistency — follows established two-layer architecture, handler pipeline, materialize pattern
2. Persistence-agnostic — works with in-memory, JSON files, SQL, event stores
3. F# idiom fit — DUs, pure functions, immutable data, computation expressions
4. Minimal domain intrusion — pure business logic modules remain untouched
5. Web API readiness — supports 202 Accepted + eventual consistency model
6. Incremental adoptability — can be introduced without rewriting all handlers
7. Testability — can be verified with existing Expecto/CE-based spec patterns
8. External integration extensibility — accommodates webhooks, message queues, third-party APIs

---

Approaches

Approach A: Atomic Outbox via Unit-of-Work Dependency

Summary: Replace the sequential materialize with a unit-of-work that intercepts Save/Publish/Dispatch calls using mutable capture variables, then commits state + outbox messages atomically.

Type definitions:

// Outbox message persisted alongside state
type OutboxMessage =
    { Id: Guid
      OccuredOn: DateTimeOffset
      MessageType: string
      Payload: string
      Destination: string }

type AtomicCommit = RondelState option -> OutboxMessage list -> Async<Result<unit, string>>

Implementation approach: A processWithOutbox wrapper at the host layer creates "capturing" dependency implementations that record calls into mutable lists, then commits everything atomically after the handler completes.

let processWithOutbox
    (store: IPersistenceAdapter)
    (uow: IUnitOfWork)
    (handler: RondelDependencies -> 'cmd -> Async<unit>)
    (cmd: 'cmd)
    : Async<unit> =
    async {
        let mutable pendingState = None
        let mutable pendingMessages = []

        let capturingDeps: RondelDependencies =
            { Load = store.Load
              Save = fun state -> async { pendingState <- Some state; return Ok() }
              Publish = fun event ->
                  async { pendingMessages <- (toOutboxMessage "event-bus" event) :: pendingMessages }
              Dispatch = fun command ->
                  async { pendingMessages <- (toOutboxMessage "accounting" command) :: pendingMessages; return Ok() } }

        do! handler capturingDeps cmd
        let! result = uow.Commit pendingState (List.rev pendingMessages)
        match result with
        | Error e -> return failwith $"Failed to commit: {e}"
        | Ok() -> ()
    }

Assessment:

Dimension	Rating	Notes
Architecture consistency	6/10	Domain materialize remains but deps become stubs. Two materialization paths.
Persistence-agnostic	8/10	UoW abstraction works with any backend
F# idiom fit	5/10	Mutable capture state, imperative UoW pattern is OOP-flavored
Minimal domain intrusion	6/10	Domain code unchanged but semantics of deps silently change
Web API readiness	8/10	Full outbox support
Incremental adoptability	5/10	Every host must adopt UoW wrapper
Testability	7/10	Domain tests unchanged; UoW independently testable
Extensibility	8/10	New destinations = new outbox message mappings

Confidence: Medium — works but feels heavy, not idiomatic F#.

---

Approach B: Functional Effect Accumulator with Atomic Commit

Summary: Replace materialize with handlers that return a RondelSideEffects record (effects as data). A single infrastructure function atomically commits everything. Pure functional approach with no mutable state.

Type definitions:

type RondelSideEffects =
    { State: RondelState option
      Events: RondelEvent list
      OutboundCommands: RondelOutboundCommand list }

// Modified handler signatures:
val execute: LoadRondelState -> RondelCommand -> Async<RondelSideEffects>
val handle: LoadRondelState -> RondelInboundEvent -> Async<RondelSideEffects>

// Infrastructure commit strategies:
type CommitRondelEffects = RondelSideEffects -> Async<Result<unit, string>>

Implementation approach: Handlers take only LoadRondelState (not full RondelDependencies) and return effects as data. The host layer chooses a commit strategy.

// Domain handler — returns effects as data
let internal move (load: LoadRondelState) (command: MoveCommand) : Async<RondelSideEffects> =
    async {
        let! state = load command.GameId
        let newState, events, commands = Move.execute command state
        return { State = newState; Events = events; OutboundCommands = commands }
    }

// Infrastructure — direct commit (terminal)
let commitDirectly save publish dispatch effects = ...

// Infrastructure — outbox commit (web)
let commitViaOutbox atomicCommit toOutboxMessages effects = ...

Assessment:

Dimension	Rating	Notes
Architecture consistency	7/10	Evolves handler pattern naturally; breaks existing materialize + RondelDependencies
Persistence-agnostic	9/10	Commit strategy is pluggable
F# idiom fit	9/10	Effects as data, pure functions, no mutable state — most idiomatic
Minimal domain intrusion	4/10	Handler signatures change, materialize removed, RondelDependencies simplified
Web API readiness	8/10	Full outbox support
Incremental adoptability	4/10	All handlers + all tests must change simultaneously
Testability	9/10	Pure data testing — no IO mocking needed
Extensibility	8/10	New commit strategies plug in without domain changes

Confidence: High for design quality, but highest migration cost.

---

Approach C: Middleware-Based Outbox (Transparent to Domain) — RECOMMENDED

Summary: Keep domain code completely unchanged. Introduce an infrastructure middleware that wraps handler execution, intercepts Save/Publish/Dispatch calls, buffers them, and commits atomically after the handler completes.

Type definitions:

module OutboxMiddleware =

    [<RequireQualifiedAccess>]
    type CapturedEffect =
        | SaveState of RondelState
        | PublishEvent of RondelEvent
        | DispatchCommand of RondelOutboundCommand

    type CapturedExecution = { Effects: CapturedEffect list }

    type AtomicCommit = RondelState option -> OutboxMessage list -> Async<Result<unit, string>>

Function signatures:

/// Execute a handler while capturing all side effects.
val captureEffects:
    LoadRondelState ->
    (RondelDependencies -> 'cmd -> Async<unit>) ->
    'cmd ->
    Async<CapturedExecution>

/// Commit captured effects atomically via outbox (web).
val commitCaptured:
    AtomicCommit ->
    (CapturedEffect -> OutboxMessage option) ->
    CapturedExecution ->
    Async<Result<unit, string>>

/// Commit captured effects directly (terminal — preserves current behavior).
val commitDirect:
    SaveRondelState ->
    PublishRondelEvent ->
    DispatchOutboundCommand ->
    CapturedExecution ->
    Async<Result<unit, string>>

Implementation:

module OutboxMiddleware =

    [<RequireQualifiedAccess>]
    type CapturedEffect =
        | SaveState of RondelState
        | PublishEvent of RondelEvent
        | DispatchCommand of RondelOutboundCommand

    type CapturedExecution = { Effects: CapturedEffect list }

    let captureEffects
        (load: LoadRondelState)
        (handler: RondelDependencies -> 'cmd -> Async<unit>)
        (cmd: 'cmd)
        : Async<CapturedExecution> =
        async {
            let captured = System.Collections.Generic.List<CapturedEffect>()

            let capturingDeps: RondelDependencies =
                { Load = load
                  Save = fun state ->
                      async {
                          captured.Add(CapturedEffect.SaveState state)
                          return Ok()
                      }
                  Publish = fun event ->
                      async { captured.Add(CapturedEffect.PublishEvent event) }
                  Dispatch = fun command ->
                      async {
                          captured.Add(CapturedEffect.DispatchCommand command)
                          return Ok()
                      } }

            do! handler capturingDeps cmd
            return { Effects = captured |> Seq.toList }
        }

    let private extractState effects =
        effects
        |> List.tryPick (function
            | CapturedEffect.SaveState s -> Some s
            | _ -> None)

    let commitCaptured
        (atomicCommit: AtomicCommit)
        (toOutboxMsg: CapturedEffect -> OutboxMessage option)
        (execution: CapturedExecution)
        : Async<Result<unit, string>> =
        let state = extractState execution.Effects
        let messages = execution.Effects |> List.choose toOutboxMsg
        atomicCommit state messages

    let commitDirect
        (save: SaveRondelState)
        (publish: PublishRondelEvent)
        (dispatch: DispatchOutboundCommand)
        (execution: CapturedExecution)
        : Async<Result<unit, string>> =
        async {
            for effect in execution.Effects do
                match effect with
                | CapturedEffect.SaveState state ->
                    let! result = save state
                    match result with
                    | Error e -> return Error e
                    | Ok() -> ()
                | CapturedEffect.PublishEvent event -> do! publish event
                | CapturedEffect.DispatchCommand command ->
                    let! result = dispatch command
                    match result with
                    | Error e -> return Error e
                    | Ok() -> ()

            return Ok()
        }

Host wiring — terminal (backward compatible):

let create (store: RondelStore) (bus: IBus) (dispatchToAccounting: DispatchToAccounting) : RondelHost =
    let dispatch outbound = toAccountingCommand outbound |> dispatchToAccounting ()
    let commit = OutboxMiddleware.commitDirect store.Save (bus.Publish: RondelEvent -> Async<unit>) dispatch

    let processMessage =
        function
        | Command cmd ->
            async {
                let! execution = OutboxMiddleware.captureEffects store.Load (execute) cmd
                let! result = commit execution
                match result with
                | Error e -> failwith $"Failed: {e}"
                | Ok() -> ()
            }
        | InboundEvent evt ->
            async {
                let! execution = OutboxMiddleware.captureEffects store.Load (handle) evt
                let! result = commit execution
                match result with
                | Error e -> failwith $"Failed: {e}"
                | Ok() -> ()
            }

    (* rest unchanged *)

Host wiring — web (outbox-backed):

let createWeb (store: IStore) (atomicCommit: AtomicCommit) : RondelHost =
    let toOutboxMsg =
        function
        | CapturedEffect.SaveState _ -> None
        | CapturedEffect.PublishEvent event ->
            Some { Id = Guid.NewGuid(); OccuredOn = DateTimeOffset.UtcNow
                   MessageType = RondelEvent.typeName event
                   Payload = JsonSerializer.Serialize event; Destination = "event-bus" }
        | CapturedEffect.DispatchCommand command ->
            Some { Id = Guid.NewGuid(); OccuredOn = DateTimeOffset.UtcNow
                   MessageType = RondelOutboundCommand.typeName command
                   Payload = JsonSerializer.Serialize command; Destination = "accounting" }

    let commit = OutboxMiddleware.commitCaptured atomicCommit toOutboxMsg
    (* same processMessage pattern as above *)

Outbox relay (background process):

module OutboxRelay =
    let start
        (loadPending: int -> Async<OutboxMessage list>)
        (acknowledge: Guid -> Async<unit>)
        (deliver: OutboxMessage -> Async<Result<unit, string>>)
        (interval: TimeSpan)
        =
        let rec loop () =
            async {
                let! pending = loadPending 50
                for msg in pending do
                    let! result = deliver msg
                    match result with
                    | Ok() -> do! acknowledge msg.Id
                    | Error _ -> ()
                do! Async.Sleep(int interval.TotalMilliseconds)
                return! loop ()
            }
        Async.Start(loop ())

Test sketch:

// All 95 existing domain tests pass without modification.

// New infrastructure tests:
testList "OutboxMiddleware" [
    testCase "captureEffects records save, publish, and dispatch" <| fun () ->
        let execution =
            OutboxMiddleware.captureEffects
                (fun _ -> async { return Some testState })
                move
                testMoveCmd
            |> Async.RunSynchronously
        Expect.isTrue
            (execution.Effects |> List.exists (function CapturedEffect.SaveState _ -> true | _ -> false))
            "captured save"

    testCase "commitCaptured writes state + outbox atomically" <| fun () ->
        let mutable committed = None
        let atomicCommit state msgs = async { committed <- Some(state, msgs); return Ok() }
        let execution = { Effects = [ CapturedEffect.SaveState testState; CapturedEffect.PublishEvent testEvent ] }
        OutboxMiddleware.commitCaptured atomicCommit toOutboxMsg execution |> Async.RunSynchronously
        Expect.isSome (fst committed.Value) "state committed"
        Expect.equal (snd committed.Value).Length 1 "one outbox message"

    testCase "commitDirect preserves current sequential behavior" <| fun () ->
        let mutable saved = false
        let mutable published = []
        let save _ = async { saved <- true; return Ok() }
        let publish evt = async { published <- evt :: published }
        let dispatch _ = async { return Ok() }
        let execution = { Effects = [ CapturedEffect.SaveState testState; CapturedEffect.PublishEvent testEvent ] }
        OutboxMiddleware.commitDirect save publish dispatch execution |> Async.RunSynchronously
        Expect.isTrue saved "state saved"
        Expect.equal published.Length 1 "event published"
]

Assessment:

Dimension	Rating	Notes
Architecture consistency	9/10	Domain completely untouched. Middleware is additive infrastructure.
Persistence-agnostic	9/10	AtomicCommit function type works with any backend
F# idiom fit	7/10	Scoped mutable List<> inside captureEffects; external API is functional
Minimal domain intrusion	10/10	Zero domain changes — materialize, RondelDependencies, handlers all unchanged
Web API readiness	8/10	Full outbox support with relay
Incremental adoptability	9/10	Terminal uses commitDirect (identical behavior). Web uses commitCaptured. Per-BC adoption.
Testability	9/10	All existing tests pass. Middleware independently testable.
Extensibility	8/10	New CapturedEffect cases, new commit strategies, new destinations

Confidence: High — lowest risk, lowest migration cost, full backward compatibility.

---

Approach D: Event-Sourced Outbox (Append-Only Effects Log)

Summary: Persist events as an append-only log instead of state snapshots. The event log serves dual duty: state reconstruction + reliable delivery (the log IS the outbox).

Implementation approach: Replace state-based persistence with event streams. Reconstruct state by folding events. Relay unpublished entries from the log.

Assessment:

Dimension	Rating	Notes
Architecture consistency	2/10	Fundamental paradigm shift from state-based to event-sourced
Persistence-agnostic	7/10	Needs append-only log abstraction
F# idiom fit	8/10	Event sourcing is natural in F# with folds and DUs
Minimal domain intrusion	2/10	Every handler, state model, and persistence adapter changes
Web API readiness	8/10	Excellent event-driven foundation
Incremental adoptability	1/10	All-or-nothing paradigm shift
Testability	7/10	Event streams are testable but all tests must be rewritten
Extensibility	9/10	Projections, temporal queries, audit trails for free

Confidence: Low — elegant but wrong fit for this codebase at this stage.

---

Comparison Matrix

Criterion (rank)	A: UoW Outbox	B: Effect Accumulator	C: Middleware Outbox	D: Event Sourced
1. Architecture consistency	6/10	7/10	9/10	2/10
2. Persistence-agnostic	8/10	9/10	9/10	7/10
3. F# idiom fit	5/10	9/10	7/10	8/10
4. Minimal domain intrusion	6/10	4/10	10/10	2/10
5. Web API readiness	8/10	8/10	8/10	8/10
6. Incremental adoptability	5/10	4/10	9/10	1/10
7. Testability	7/10	9/10	9/10	7/10
8. Extensibility	8/10	8/10	8/10	9/10
Weighted total	6.4	7.1	8.8	5.0

---

Recommendation

Approach C: Middleware-Based Outbox is the recommended strategy.

Justification

1. Zero domain changes — materialize, RondelDependencies, all handler signatures, and all 95 existing tests remain completely untouched
2. Backward compatible — terminal app uses commitDirect with behavior identical to today's sequential materialize
3. Web-ready — swap to commitCaptured with atomic outbox commit for persistent deployments
4. Incremental adoption — can be introduced per bounded context independently; Accounting follows the same pattern when ready
5. Low risk — infrastructure-only change, domain logic fully protected from regression
6. Extensible — new destinations (webhooks, message queues, external APIs) are just new toOutboxMsg mappings in the host layer

Why Not the Others

• Approach A achieves similar results but with less idiomatic F# (mutable UoW pattern) and higher conceptual overhead
• Approach B is the most elegant functional design, but the migration cost of changing all handler signatures and tests is disproportionate to the benefit, especially when Approach C achieves the same consistency guarantees without touching domain code
• Approach D is a paradigm shift that solves problems the project doesn't yet have, at enormous migration cost

---

Risks & Mitigations

Risk	Likelihood	Impact	Mitigation
Outbox relay delivers messages out of order	Medium	Medium	Relay processes messages sequentially per stream (GameId). Include sequence numbers for consumer-side ordering verification.
Relay crashes between delivery and acknowledgment → duplicate delivery	Medium	Low	Inbox/idempotent consumer pattern on receiving side. Most handlers already have partial idempotency.
captureEffects interception misses side effects from handler reading bus during execution	Low	High	Current handlers never read from bus during execution. Add architectural test asserting handlers only use injected deps.
Scoped mutable List<> in captureEffects causes threading issues	Very Low	Medium	captureEffects runs within a single async {} CE — sequential by design. MailboxProcessor guarantees single-threaded processing.
Outbox table grows unbounded	Medium	Low	Retention policy: move acknowledged messages to archive table or delete after configurable TTL.
Serialization/deserialization of DU types for outbox payload	Medium	Medium	Use System.Text.Json with F# support or Thoth.Json. Test roundtrip serialization for all event/command types.

---

Migration Path

Step 1: Introduce OutboxMiddleware module (infrastructure only)

• Create OutboxMiddleware.fs in a shared infrastructure location (e.g., src/Imperium.Infrastructure/ or within src/Imperium.Terminal/)
• Implement captureEffects, commitDirect, commitCaptured
• Write middleware unit tests
• Validation: dotnet build && dotnet test — all 95 tests pass

Step 2: Wire terminal app through middleware with commitDirect

• Modify RondelHost.create to use captureEffects + commitDirect instead of passing deps directly to domain handlers
• Modify AccountingHost.create similarly
• Validation: dotnet build && dotnet test — all 95 tests pass, terminal app behavior identical

Step 3: Define OutboxMessage types and AtomicCommit abstraction

• Create OutboxMessage record type
• Define AtomicCommit function type
• Implement toOutboxMsg mapping functions for RondelEvent and RondelOutboundCommand
• Test serialization roundtrips
• Validation: Unit tests for serialization

Step 4: Implement persistence-specific AtomicCommit (when persistence adapter is chosen)

• Depends on Plan JSON file persistence adapter for terminal store (deferred) #75 (JSON file persistence) or future database choice
• Implement AtomicCommit for the chosen backend
• Implement outbox storage (pending + processed message tracking)
• Validation: Integration tests with actual persistence

Step 5: Implement OutboxRelay background process

• Polling-based relay that reads pending messages, delivers them, and acknowledges
• Configure polling interval (start with 1-5 seconds)
• Add delivery error logging
• Validation: Integration tests verifying reliable delivery

Step 6: Wire web app through middleware with commitCaptured

• Create web-specific host wiring using commitCaptured + AtomicCommit
• Start OutboxRelay in web app startup
• Validation: End-to-end tests with actual persistence + relay

Step 7 (follow-up): Idempotent consumer / inbox on receiving side

• Add message ID tracking to inbound event handlers
• Retrofit move and setToStartingPositions with idempotency checks where needed
• Validation: Tests verifying duplicate message rejection

---

Test Strategy

Existing Tests (unchanged)

All 95 existing tests continue to pass without modification. They exercise domain logic through the existing RondelDependencies interface, which the middleware preserves.

New Middleware Tests

Test	Verifies
captureEffects records Save calls	Effect capture works for state persistence
captureEffects records Publish calls	Effect capture works for event publication
captureEffects records Dispatch calls	Effect capture works for outbound commands
captureEffects preserves effect ordering	Save → Publish → Dispatch order maintained
commitDirect replays effects sequentially	Backward compatibility with current behavior
commitDirect fails fast on Save error	Error propagation matches current materialize
commitDirect fails fast on Dispatch error	Error propagation matches current materialize
commitCaptured commits state + messages atomically	Outbox atomicity guarantee
commitCaptured excludes SaveState from outbox messages	State committed separately from message relay
Serialization roundtrip for all RondelEvent cases	Outbox payload integrity
Serialization roundtrip for all RondelOutboundCommand cases	Outbox payload integrity

Integration Tests (when persistence is available)

Test	Verifies
AtomicCommit persists state + outbox together	Transaction atomicity
AtomicCommit rolls back both on failure	No partial commits
OutboxRelay delivers pending messages	Relay mechanism works
OutboxRelay retries on delivery failure	At-least-once guarantee
OutboxRelay acknowledges after successful delivery	No duplicate relay
Duplicate message delivery is idempotent	Inbox pattern works

[bartul]

Take: keep the outbox decision, change the implementation target

Short version: the issue correctly identifies the consistency problem and the right family of patterns. The durable/web-ready solution should be transactional outbox + idempotent inbox, not outbox alone. However, the recommended implementation should shift away from transparent middleware capture as the target architecture. The long-term target should be an explicit effect commit boundary: domain execution produces state/events/outbound commands as data, and infrastructure chooses how to commit those effects.

Middleware capture remains a possible compatibility bridge, but it should not be the design we optimize toward.

Grounded observations from the current code

The current consistency boundary is Rondel.materialize:

let internal materialize deps state events commands : Async<unit> =
    async {
        match state with
        | Some s ->
            let! result = deps.Save s
            match result with
            | Error e -> return failwith $"Failed to save state: {e}"
            | Ok() -> ()
        | None -> ()

        for event in events do
            do! deps.Publish event

        for command in commands do
            let! result = deps.Dispatch command
            match result with
            | Error e -> return failwith $"Failed to dispatch command: {e}"
            | Ok() -> ()
    }

This is a classic dual-write/multi-write boundary. If save succeeds and a later publish/dispatch fails, state has already moved forward. For paid rondel movement, this can leave a pending movement in state without a durable accounting command.

There are three important repo-specific details:

1. Terminal dispatch is currently only an enqueue acknowledgment.

RondelHost.create store bus (fun () cmd ->
    async {
        do! accountingHost.Value.Execute cmd
        return Ok()
    })

AccountingHost.Execute posts to a mailbox and returns. That means current Dispatch success means “command entered the Accounting mailbox”, not “Accounting processed payment”. This is intentional per #74, and it maps well to future HTTP 202 Accepted semantics.

2. The terminal bus deliberately swallows subscriber exceptions.

let invokeSafely handler event =
    async {
        try
            do! handler event
        with _ ->
            ()
    }

So publishing to the bus cannot prove delivery/handling. An outbox can guarantee the message is durably recorded and later attempted, but consumers still need idempotency and failure recovery.

3. Accounting is currently stateless.

type AccountingDependencies = { Publish: PublishAccountingEvent }

Until #49 introduces Accounting state/load/save, only Rondel has a meaningful state+effects atomicity problem. That makes a full outbox implementation premature right now; it would be infrastructure without a durable persistence boundary to protect.

Refined hypothesis tree

H1: Status quo is acceptable for the terminal sandbox

Confidence: high.

The terminal app is in-memory and intentionally fire-and-forget. An in-memory outbox would not survive process failure, so it would add ceremony without the core reliability guarantee. Existing supervision is enough for the current sandbox phase.

H2: Transactional outbox is needed for web/durable persistence

Confidence: high.

Once state is durable, the system must avoid “state committed, message lost.” The durable commit should atomically persist the aggregate state and outbound messages. A relay then publishes those messages.

H3: Outbox alone is sufficient

Confidence: low.

Outbox creates at-least-once delivery. The relay can publish a message and crash before marking it processed, so duplicates are expected. Consumers must be idempotent, using either processed-message records or business-state-level idempotency.

H4: Middleware capture is the best implementation

Confidence: medium-low.

Middleware capture has low migration cost, but it fakes dependency semantics. Inside the domain handler, Save, Publish, and Dispatch appear to succeed, but they are only captured. That is safe only as long as handlers never branch on those results in meaningful ways. The abstraction is too implicit for a consistency boundary.

H5: Explicit effect commit boundary is the best long-term implementation

Confidence: high.

Rondel already has the important pure shape internally:

RondelState option * RondelEvent list * RondelOutboundCommand list

The better move is to name that shape, return it from the handler pipeline, and commit it explicitly in infrastructure. That keeps the consistency boundary visible and testable.

Approach assessment

Approach A: Unit-of-work dependency

This is workable, but it feels like an OOP unit-of-work wrapper around an already functional codebase. It still tends to hide what is being committed, and it does not align as cleanly with the existing pure execution modules.

Verdict: not preferred.

Approach B: Functional effect accumulator / explicit commit boundary

This is the best target architecture. Domain command/event processing returns effects as data. Infrastructure commits them directly in terminal mode or atomically with an outbox in durable mode.

The main cost is a signature/refactor cost, but the refactor aligns with the existing architecture rather than fighting it.

Verdict: preferred long-term approach.

Approach C: Middleware-based outbox

This is the lowest migration-cost bridge, but it makes RondelDependencies lie during captured execution. It also creates two meanings for the same dependency record:

• normal execution: Save really saves, Publish really publishes, Dispatch really dispatches/enqueues;
• captured execution: these functions append to a list and return success.

That is tolerable for a tactical adapter but too implicit for the permanent architecture.

Verdict: acceptable bridge, not preferred target.

Approach D: Event sourcing

Event sourcing would make the log itself the source of truth and can naturally support replay, audit, and message relay. But this repo is currently state-snapshot based, and shifting now would rewrite too much for too little immediate benefit.

Verdict: defer indefinitely unless audit/replay becomes a real product requirement.

Recommended target architecture

Introduce a named effect type and move materialization out of the domain handler core.

type RondelEffects =
    { State: RondelState option
      Events: RondelEvent list
      Commands: RondelOutboundCommand list }

module RondelEffects =
    let none =
        { State = None
          Events = []
          Commands = [] }

    let ofTuple (state, events, commands) =
        { State = state
          Events = events
          Commands = commands }

Then separate domain execution from committing:

type LoadRondelState = Id -> Async<RondelState option>

type CommitRondelEffects = RondelEffects -> Async<Result<unit, string>>

type RondelExecutionDependencies =
    { Load: LoadRondelState
      Commit: CommitRondelEffects }

Handlers become “load -> pure execute -> commit effects”:

let internal move (deps: RondelExecutionDependencies) (command: MoveCommand) : Async<unit> =
    async {
        let! state = deps.Load command.GameId

        let effects =
            Move.execute command state
            |> RondelEffects.ofTuple

        let! result = deps.Commit effects

        match result with
        | Ok () -> ()
        | Error e -> failwith $"Failed to commit rondel effects: {e}"
    }

Public API can remain execute : RondelDependencies -> RondelCommand -> Async<unit> temporarily by adapting old dependencies into a direct commit function:

let commitDirect (deps: RondelDependencies) (effects: RondelEffects) : Async<Result<unit, string>> =
    async {
        match effects.State with
        | Some state ->
            let! saved = deps.Save state
            match saved with
            | Error e -> return Error e
            | Ok () -> ()
        | None -> ()

        for event in effects.Events do
            do! deps.Publish event

        for command in effects.Commands do
            let! dispatched = deps.Dispatch command
            match dispatched with
            | Error e -> return Error e
            | Ok () -> ()

        return Ok ()
    }

That gives a safe migration path: first name the effects and commit function, then later introduce an outbox-backed commit without changing pure domain execution.

Web/durable commit sketch

A durable outbox commit should persist state and messages in one transaction.

type OutboxMessageId = OutboxMessageId of Guid

type OutboxDestination =
    | DomainEventBus
    | AccountingCommandQueue

type OutboxMessage =
    { Id: OutboxMessageId
      GameId: Id
      Destination: OutboxDestination
      MessageType: string
      Payload: string
      OccurredAt: DateTimeOffset
      Sequence: int64 }

type AtomicRondelCommit =
    RondelState option -> OutboxMessage list -> Async<Result<unit, string>>

The outbox-backed commit maps effects to messages and persists them with the state:

let commitWithOutbox
    (serializeEvent: RondelEvent -> string)
    (serializeCommand: RondelOutboundCommand -> string)
    (nextSequence: unit -> int64)
    (atomicCommit: AtomicRondelCommit)
    (effects: RondelEffects)
    : Async<Result<unit, string>> =

    let now = DateTimeOffset.UtcNow

    let gameIdFromEvent event =
        match event with
        | PositionedAtStart e -> e.GameId
        | ActionDetermined e -> e.GameId
        | MoveToActionSpaceRejected e -> e.GameId

    let gameIdFromCommand command =
        match command with
        | ChargeMovement c -> c.GameId
        | VoidCharge c -> c.GameId

    let eventMessages =
        effects.Events
        |> List.map (fun event ->
            { Id = OutboxMessageId(Guid.NewGuid())
              GameId = gameIdFromEvent event
              Destination = DomainEventBus
              MessageType = event.GetType().Name
              Payload = serializeEvent event
              OccurredAt = now
              Sequence = nextSequence () })

    let commandMessages =
        effects.Commands
        |> List.map (fun command ->
            { Id = OutboxMessageId(Guid.NewGuid())
              GameId = gameIdFromCommand command
              Destination = AccountingCommandQueue
              MessageType = command.GetType().Name
              Payload = serializeCommand command
              OccurredAt = now
              Sequence = nextSequence () })

    atomicCommit effects.State (eventMessages @ commandMessages)

The important point: atomicCommit must be the only place that knows the persistence technology. For SQL, it is one database transaction. For Cosmos-like stores, it must respect partition/transaction constraints. For JSON file storage, it must be a careful file replacement strategy, but JSON file persistence is probably not worth using as the first serious outbox backend.

Relay sketch

Relay is at-least-once by design.

type LoadPendingOutboxMessages = int -> Async<OutboxMessage list>
type MarkOutboxMessageProcessed = OutboxMessageId -> Async<Result<unit, string>>
type DeliverOutboxMessage = OutboxMessage -> Async<Result<unit, string>>

let relayOnce
    (loadPending: LoadPendingOutboxMessages)
    (deliver: DeliverOutboxMessage)
    (markProcessed: MarkOutboxMessageProcessed)
    : Async<unit> =
    async {
        let! messages = loadPending 100

        for message in messages |> List.sortBy (fun m -> m.Sequence) do
            let! delivered = deliver message

            match delivered with
            | Error _ ->
                // Leave pending. Retry later.
                ()
            | Ok () ->
                let! marked = markProcessed message.Id

                match marked with
                | Ok () -> ()
                | Error _ ->
                    // Message may be delivered again. Consumers must be idempotent.
                    ()
    }

A production relay would also need backoff, attempt counts, dead-lettering/poison-message handling, observability, and likely per-aggregate ordering. For this repo, ordering per GameId is probably the correct first constraint.

Inbox/idempotency sketch

Outbox without inbox is incomplete. At-least-once delivery means duplicates are normal.

type MessageId = MessageId of Guid

type InboxConsumer =
    | RondelHost
    | AccountingHost

type TryStartMessageProcessing =
    InboxConsumer -> MessageId -> Async<Result<bool, string>>

type MarkMessageProcessed =
    InboxConsumer -> MessageId -> Async<Result<unit, string>>

A generic helper can wrap inbound processing:

let handleInboxMessage
    (consumer: InboxConsumer)
    (messageId: MessageId)
    (tryStart: TryStartMessageProcessing)
    (markProcessed: MarkMessageProcessed)
    (process: unit -> Async<unit>)
    : Async<unit> =
    async {
        let! startResult = tryStart consumer messageId

        match startResult with
        | Error e -> failwith $"Failed to start inbox processing: {e}"
        | Ok false ->
            // Already processed or currently owned by this consumer.
            ()
        | Ok true ->
            do! process ()

            let! markResult = markProcessed consumer messageId
            match markResult with
            | Ok () -> ()
            | Error e -> failwith $"Failed to mark inbox message processed: {e}"
    }

For some handlers, business idempotency already exists. For example, Rondel invoice payment events with no matching pending movement are ignored. That helps. But stateful Accounting debits will not be naturally idempotent if the same charge message is delivered twice. Accounting should track billing/message IDs before subtracting balance.

What this means for #49

Stateful Accounting should be designed with idempotency from the start.

For example:

type AccountingState =
    { GameId: Id
      Balances: Map<string, Amount>
      ProcessedCharges: Set<Id> }

Charge processing can then be idempotent by billing ID:

let applyCharge cmd state =
    if state.ProcessedCharges |> Set.contains cmd.BillingId then
        state, []
    else
        match state.Balances |> Map.tryFind cmd.Nation with
        | None ->
            state,
            [ RondelInvoicePaymentFailed { GameId = cmd.GameId; BillingId = cmd.BillingId } ]
        | Some balance when Amount.value balance >= Amount.value cmd.Amount ->
            let newBalance = Amount.unsafe (Amount.value balance - Amount.value cmd.Amount)

            { state with
                Balances = state.Balances |> Map.add cmd.Nation newBalance
                ProcessedCharges = state.ProcessedCharges |> Set.add cmd.BillingId },
            [ RondelInvoicePaid { GameId = cmd.GameId; BillingId = cmd.BillingId } ]
        | Some _ ->
            { state with ProcessedCharges = state.ProcessedCharges |> Set.add cmd.BillingId },
            [ RondelInvoicePaymentFailed { GameId = cmd.GameId; BillingId = cmd.BillingId } ]

Whether failed charges should be marked processed is a business decision. If insufficient funds can later become sufficient and retry should succeed, then do not mark failure as final. If each invoice is a one-time decision, mark it processed and require a new billing ID for a new attempt. The current Rondel model treats a billing ID as a single invoice attempt, so marking terminal failure as processed is consistent.

Migration path

Phase 0: Keep current terminal behavior

No immediate code change is necessary for the in-memory terminal app. The current system is intentionally non-durable and supervised.

Phase 1: Implement stateful Accounting (#49) with idempotency

Add state/load/save and processed billing IDs or processed message IDs. This makes the consumer side safe before introducing retrying delivery.

Phase 2: Name the effect boundary in Rondel

Introduce RondelEffects and CommitRondelEffects. Replace materialize with commitDirect as an explicit adapter.

This phase can preserve the existing public API:

let execute (deps: RondelDependencies) (cmd: RondelCommand) : Async<unit> =
    let executionDeps =
        { Load = deps.Load
          Commit = commitDirect deps }

    executeWithCommit executionDeps cmd

Phase 3: Add durable outbox types and storage

Add outbox message identity, destination, payload, sequence, attempt count, and processed/dead-letter metadata. Choose the durable backend first; do not implement this seriously on top of in-memory storage.

Phase 4: Add relay and inbox

The relay delivers pending outbox messages. Receiving hosts wrap inbound message handling with inbox/idempotency. Add explicit duplicate delivery tests.

Phase 5: Switch web host to outbox commit

The web host uses commitWithOutbox; terminal can continue using commitDirect.

Test strategy

Add tests at the level where the guarantee exists.

Effect boundary tests

• Move.execute still returns expected state/events/commands.
• commitDirect saves before publishing and dispatching.
• commitDirect fails fast on save error and publishes nothing.
• commitDirect reports dispatch failure after save, preserving current terminal behavior.

Outbox commit tests

• State and outbox messages are committed atomically.
• Save failure writes neither state nor messages.
• Event and command messages include stable IDs, destination, type, payload, GameId, and sequence.
• Messages are ordered per game.

Relay tests

• Pending messages are delivered in sequence.
• Failed delivery remains pending.
• Successful delivery followed by failed mark-processed causes duplicate delivery on retry.

Inbox/idempotency tests

• Duplicate RondelInvoicePaid is ignored.
• Duplicate Accounting charge does not debit twice.
• Duplicate void charge is safe.
• Failed charge retry behavior matches the chosen business rule.

Updated recommendation for this issue

The issue can be closed once the decision is recorded as:

Adopt transactional outbox + idempotent inbox for durable/web deployments. Do not implement it for the current in-memory terminal sandbox. When implementation begins, prefer an explicit effect commit boundary over transparent middleware capture. Middleware capture is acceptable only as a tactical compatibility bridge.

The concrete follow-up work should be separate issues:

1. Make Accounting stateful and idempotent (Implement Stateful Accounting Domain with Balance Tracking #49 already covers most of this; it should explicitly include duplicate charge handling).
2. Introduce RondelEffects / CommitRondelEffects and preserve terminal direct commit behavior.
3. Design durable outbox storage and relay for the chosen persistence backend.
4. Add inbox/processed-message tracking for inbound messages and Accounting charges.

This keeps the architecture honest: domain logic remains explicit and testable, terminal stays simple, and the web phase gets the real guarantee rather than a middleware abstraction that only appears atomic.

[bartul]

Updated the tracking records for the gaps found in the review:

• Plan transactional outbox/retry strategy for save-publish-dispatch consistency (deferred) #76 now records the final decision in the issue body, including outbox + idempotent inbox, the terminal sandbox non-goal, and the explicit effect commit boundary target.
• The failure matrix now clarifies that terminal bus publish success does not prove subscriber handling because subscriber exceptions are suppressed.
• Implement Stateful Accounting Domain with Balance Tracking #49 now explicitly requires Accounting BillingId idempotency for duplicate charge and void command delivery.
• Follow-up implementation issues were created:
  • type = improvement: Introduce explicit Rondel effect commit boundary #129 - explicit Rondel effect commit boundary
  • type = improvement: Design durable outbox storage and relay #130 - durable outbox storage and relay
  • type = improvement: Add inbox and processed-message tracking for inbound delivery #131 - inbox and processed-message tracking