Skip to content

Conversation

@polRk
Copy link
Member

@polRk polRk commented Nov 6, 2025

What

Fixed critical seqNo renumbering bug in TopicWriter and simplified API by removing return value from write() method.

Why

Bug fix:
Previously, when messages were written before session initialization (before receiving lastSeqNo from server), auto-generated seqNo started from 0 and were never renumbered after session initialization. This caused seqNo conflicts when:

  • Multiple writers used the same producerId
  • Writer reconnected after network issues
  • Messages were written immediately after writer creation

The fix ensures that all pending messages in auto seqNo mode are properly renumbered to continue from server's lastSeqNo + 1 after session initialization.

API simplification:
The write() method previously returned seqNo values that could be misleading - they were temporary before session initialization and could change after reconnection. Removing the return value simplifies the API and prevents misuse.

Changes

Bug Fix: seqNo Renumbering

  • machine.ts: Implemented proper seqNo renumbering in updateWriteSession action
    • In auto seqNo mode: messages written before session initialization are now renumbered sequentially starting from serverLastSeqNo + 1
    • In manual seqNo mode: user-provided seqNo remain unchanged (as before)
    • Properly handles sliding window compaction and message renumbering

API Changes

  • writer.ts:
    • write() method now returns void instead of bigint (removed seqNo return value)
    • Added isSessionInitialized flag to track session state
    • Updated writer.session event handling to use nextSeqNo from state machine
    • Added seqNoMode parameter to writer.write event for state machine

Code Improvements

  • types.ts: Updated WriterEmitted type to include nextSeqNo in writer.session event
  • seqno-manager.ts: Enhanced to support mode detection and state tracking
  • Tests: Updated to remove assertions on write() return values, verify seqNo renumbering works correctly

Migration Guide

If you were storing seqNo from write() return value:

// Before
let seqNo = writer.write(data)
await saveToDatabase(seqNo)

// After
writer.write(data)
let lastSeqNo = await writer.flush() // Get final seqNo after flush
await saveToDatabase(lastSeqNo)Important:

  • User-provided seqNo (via extra.seqNo) remain final and unchanged - no migration needed
  • After flush() completes, all seqNo values up to returned lastSeqNo are final
  • Use sequential order of write() calls to track individual messages if needed

Testing

  • ✅ Added tests verifying seqNo renumbering works correctly during reconnections
  • ✅ Updated existing tests to work with new API (no return value from write())
  • ✅ Verified messages are properly sequenced after session initialization
  • ✅ All existing tests pass

Files Changed

  • packages/topic/src/writer2/machine.ts - Implemented seqNo renumbering logic
  • packages/topic/src/writer2/writer.ts - Removed return value, added session tracking
  • packages/topic/src/writer2/types.ts - Updated event types
  • packages/topic/src/writer2/seqno-manager.ts - Enhanced seqNo management
  • packages/topic/tests/writer2.test.ts - Updated tests
  • .changeset/remove-write-seqno-return.md - Added changeset

Checklist

  • ✅ Changeset added
  • ✅ Tests updated
  • ✅ Linter passes
  • ✅ Build passes

@nikolaymatrosov
Copy link

If the seqNo returned by write() cannot be reliably used until the session is initialized or flushed, perhaps it would be safer not to return a value that looks like a final sequence number at all.

Instead, we could return an opaque object (e.g. a SeqToken) that encapsulates the temporary state and provides a method like resolveSeqNo() to obtain the final value later. This approach would prevent users from mistakenly persisting or relying on a non-final seqNo and make the API usage pattern more explicit.

interface SeqToken {
  resolveSeqNo(): bigint
}

class TopicWriter {
  write(
    data: Uint8Array,
    extra?: {
      seqNo?: bigint
      createdAt?: Date
      metadataItems?: Record<string, Uint8Array>
    }
  ): SeqToken {
    ...
  }
}

@polRk
Copy link
Member Author

polRk commented Nov 7, 2025

it would be safer not to return a value that looks like

I thought about it, but it's a loss of backward compatibility.

write(): SeqToken

I intentionally did not make the return value an object so that there would not be many references and the GC could dispose of memory more efficiently.

polRk added 2 commits November 7, 2025 19:37
- Add resolveSeqNo() method to get final seqNo for messages written before session initialization
- Track seqNo shifts through SeqNoShiftEvent segments when session reconnects
- Update write() JSDoc to warn about temporary seqNo values before session initialization
- Implement efficient seqNo shift tracking using range merging and inversion algorithms
- Update flush() documentation to clarify when seqNo values become final

Breaking changes: None (backward compatible)
- Extract seqNo mapping logic into SeqNoResolver class for better testability
- Add SeqNoShiftBuilder to accumulate shifts during message renumbering
- Simplify manual seqNo mode: only adjust pointers, no array mutations
- Add comprehensive unit tests for SeqNoResolver, SeqNoShiftBuilder, SeqNoManager
- Improve code documentation and comments throughout updateWriteSession
- Add integration test verifying resolveSeqNo works correctly
@polRk polRk force-pushed the feature/topic-writer-resolve-seqno branch from a4e6fbf to 3873268 Compare November 7, 2025 16:37
- Fix seqNo renumbering bug: messages written before session initialization are now properly renumbered after receiving lastSeqNo from server
- Remove return value from write() method (now returns void) to simplify API
- Remove resolveSeqNo() method and related seqNo shift tracking infrastructure
- Update tests to remove assertions on write() return values
- Add changeset describing bug fix and API simplification
@polRk polRk force-pushed the feature/topic-writer-resolve-seqno branch 3 times, most recently from 3917974 to b0ee2e9 Compare November 7, 2025 18:48
…tion

- Remove lastSeqNo update from _flush() - it was updating before messages were actually sent to server
- Fix renumbering logic in _on_init_response to properly handle messages written before init
- Check if messages in buffer need renumbering by comparing their seqNo with serverLastSeqNo
- Never renumber messages if user provided seqNo (manual mode)
- Update lastSeqNo only when session is initialized or new message is written, not on ACKs
@polRk polRk force-pushed the feature/topic-writer-resolve-seqno branch from b0ee2e9 to a33d8b1 Compare November 7, 2025 18:53
@polRk polRk changed the title Add resolveSeqNo method and track temporary seqNo shifts in TopicWriter Fixed critical seqNo renumbering bug in TopicWriter Nov 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants