Skip to content

Add opt-in Wired Memory - #6

Open
ronaldmannak wants to merge 5 commits into
Layr-Labs:mainfrom
PicoMLX:layr/wiredmemory
Open

Add opt-in Wired Memory#6
ronaldmannak wants to merge 5 commits into
Layr-Labs:mainfrom
PicoMLX:layr/wiredmemory

Conversation

@ronaldmannak

Copy link
Copy Markdown

Add an opt-in Wired Memory execution boundary for continuous batching.

BatchGenerator now has a drain(wiredMemoryTicket:onResponse:) convenience method that holds a WiredMemoryTicket across the full drain loop, rather than starting and ending the ticket for each next() step. The low-level next() API remains synchronous.

Changes

  • Added BatchGenerator.drain(wiredMemoryTicket:onResponse:).
  • Kept the ticket active until all queued/active rows are drained, the response handler throws, or the task is cancelled.
  • Added cancellation checks before each generator step and before each response callback.
  • Documented partial-drain behavior: delivered responses are not replayed or rolled back.
  • Documented the single-driver contract: do not call insert, cancel, next, or close concurrently with drain.
  • Added DocC guidance for bounded batches and long-running server loops.
  • Added helper-level tests proving the drain scope wraps the full loop and cancellation stops before the next step.

Notes

Ticket sizing remains caller-owned in this PR. Automatic continuous-batch memory estimation is intentionally left for a future scheduler/container layer because it depends on cache topology, prompt shape, completionBatchSize, and admission policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant