Skip to content

Upstream tokenspeed: msgspec/msgpack IPC mode for io_struct #2005

Description

@slin1237

Part of #2000; gates #2006.

TokenSpeed's frontend↔scheduler IPC is sglang topology (EngineCoreClient PUSH/PULL over PortArgs) but tokenspeed/runtime/engine/io_struct.py is plain-dataclass, pickle-only — no msgpack mode, so no non-Python frontend can speak it.

Upstream PR to lightseekorg/tokenspeed, using sglang 0.5.16's io_struct migration as the template:

  • Port io_struct dataclasses to kw-only msgspec Structs (tag=True, array_like=True base, matching sglang's BaseReq pattern)
  • sock_send/sock_recv helpers honoring a pickle/msgpack env toggle (pickle stays default, like SGLANG_USE_PICKLE_IPC)
  • Adopt typed msgspec multimodal structs from day one (the Upstream sglang: typed msgspec multimodal structs for msgpack IPC #2004 design) — skip the PickleWrapper intermediate step
  • Update AsyncLLM/EngineCoreClient/scheduler send-recv paths to the helpers
  • Tests: round-trip parity pickle vs msgpack for generate/embedding/abort/output structs

Lessons from the sglang 0.5.16 consumer-side port (smg #1970) worth baking in: slotted structs turn phantom-attribute writes into hard errors (audit setattr sites); msgspec reports missing required args one at a time (make required fields deliberate); array("q") token-id containers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestzmq-backendZMQ direct backend workstream

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions