feat: engine-native image references via kimi-file:// media resolver - #2593
Open
7Sageer wants to merge 40 commits into
Open
feat: engine-native image references via kimi-file:// media resolver#25937Sageer wants to merge 40 commits into
7Sageer wants to merge 40 commits into
Conversation
🦋 Changeset detectedLatest commit: 501e86f The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…media ref contract
…ath, keep unpaired tags
…nal projection A projector that attached after turn.started built the terminal turn.upsert with an empty header, and the whole-header replace downstream wiped the backfilled origin / prompt / attachmentIds — only the debounced best-effort heal could restore them. Fall back to the producer store's seeded header (via a new optional ProjectorLookups.turn) when currentTurn misses, and cover the mid-turn attach path with a service-level regression test.
…gent/media The kimi-file:// daemon reference grammar, media path tags, and the tag/ref fold are engine-internal conventions, not provider-wire contract; keep src/kosong untouched. Root exports and SDK re-exports are unchanged.
Pasted and uploaded media now materialize under the session's own media/ dir instead of the shared cache, so the copies follow the session's lifecycle: fork carries them along, session deletion cleans them up. A new Session-scope ISessionMediaStore owns the dir: atomic tmp+rename materialization with a unified extension policy, and canonical-vs-hint display-path resolution. The persisted ?path= is a write-time snapshot — readers prefer the session-canonical location, so fork and home relocation never hand the model a dead path. Prompt intake normalizes every daemon reference through the single enqueue funnel (REST edge, SDK prompt/steer, gateway), serialized in arrival order to keep the FIFO across the async file I/O. The kap-server edge materializes through the same store with a shared-cache fallback, and the request-time resolver refreshes stale persisted and memoized path tags; a claimed video reference degrades to its tag alone instead of duplicating it.
7Sageer
marked this pull request as ready for review
August 5, 2026 05:31
# Conflicts: # apps/kimi-code/src/tui/kimi-tui.ts # packages/node-sdk/src/sdk-rpc-client-v2.ts
commit: |
… path The record now joins the FIFO synchronously and its daemon-ref intake runs as a per-record promise, awaited by the launch and steer paths before the message is consumed — queue order, list/abort visibility, and prompt submission latency no longer wait on file I/O, and a slow intake no longer head-of-line blocks later prompts. The launching record is tracked so abort and clear stay reachable inside the launch window; startNext re-checks cancellation after every await (intake race, hook, turn admission), a cancelled record is never re-queued, and a compaction requeue waits for onDidFinishCompaction instead of busy-looping the scheduler.
…ring pairMediaPathTagRefs now exposes claimingRefByTagIndex, and claimingRefIndex reads it instead of recovering the claimer by path equality — which mis-attributed a tag when two different fileIds carried the same path in an interleaved sequence, breaking the pair and leaking the tag as user text. Also covers the memoized-video-tag claimed-drop branch.
…g parity The cold rebuild's user-slash branch now folds the turn-opening input like any user turn (claimed tag out of the prompt text, one attachment entity), matching the live projection. The ref extraction is consolidated into the contract module (daemonFileRefFromPairingPart, the mirror of the engine's daemonFileRefFromPart) and the mirror carries the new claimingRefByTagIndex map. A new kap-server parity test imports both implementations and asserts identical pairings over shared fixtures, so the engine/mirror pair can no longer drift silently.
The global search indexer concatenated every text part of a persisted user message, so the upload pair's <image path> tag made pure-image prompts searchable and wrote the materialization path into the index — breaking the module's documented pure-image invariant and diverging from the live route. textOfContent now folds the pair like every other read model (with a fold-safe coercion for malformed wire parts). Also pins the prompt-media cache-dir fallback with a read-only session media dir test (skipped as root).
…oadFile rejection foldMediaPathTagRefs and matchSingleMediaPathTag join the daemon file-reference helper re-exports so hosts can fold the upload tag+ref pair without importing agent-core-v2; the v1 harness's uploadFile not_implemented rejection is pinned by a test.
…s atomic in steer input Resumed-session replay rendered the upload pair raw — the <image path> tag as user text and the kimi-file:// url as an XML-ish reference — and the markdown export leaked the tag into both the turn body and the overview topic. contentPartsToText and the exporter now fold the pair, and daemon references render as a bare [image]/[video] placeholder. combineSteerInput moves to tui/utils/steer-input and no longer merges a standalone media tag into adjacent text, which would have broken the engine-side pairing for steered image messages.
A queued pure-image message opens with a standalone `<media path>` tag, which combineSteerInput keeps atomic. With the previous item ending in a media part, the '\n\n' separator landed as a stranded whitespace-only text part between the media part and the tag, normalizePromptInput rejected the steer, and the already-cleared queue lost the messages. Treat a leading standalone tag as media so the separator is dropped there.
A daemon reference's fileId becomes a storage key in the session media store, but only the file domain validated the id shape — a crafted kimi-file://<id> reaching the request-time resolver's canonical-read fallback could traverse out of the session media dir. Share the file domain's id regex and guard every store entry point: reads miss, materialize declines, and the display path falls back to the hint.
…n refs
prompt.steered published the raw engine content parts — kimi-file://
refs carrying the absolute materialization path plus the paired
<media path> tag — to both the legacy session_event wire (whose schema
declares the protocol content shape) and the transcript prompt entity.
Route both through one shared prompt-content projection: the upload
pair folds into a single {kind:'file'} part, matching the REST prompt
list and the no-path-leak rule every sibling surface already follows.
The deprecated videoResolverService alias also re-exported mediaResolvedKey, which made the package root's star exports ambiguous and silently dropped the name. The new transcript contract mirror now uses the canonical daemon-ref vocabulary instead of the deprecated kimi-file spelling.
…rough, release-once Mirror the video abort contract on the new image path (an aborted read cancels the request instead of degrading to a tag), cover the video fallback that uploads the session-canonical bytes after the transient upload is released, and assert the staged-upload release fires exactly once on the intake success path.
sendMessageInternal read the turn context only after beginSessionRequest had cleared it, so a steer buffered into a running goal turn never got its staging lease bound — the staged daemon upload and cache copies lived until session close instead of being released at the consuming turn's end. Capture the live turn id before the reset (only while a turn is actually streaming; the id outlives its turn otherwise). Also move the staging-lease state machine off the KimiTUI coordinator into a self-contained StagingLeaseTracker with injected effects, drop the duplicate media-tag builder in image-placeholder in favor of the SDK helper, and fix the paste-in-flight comment to match the gate's real granularity.
… refs
The broadcaster projected prompt.steered and stripped turn.started
attachments but forwarded prompt.queued raw, leaking kimi-file:// URLs
and absolute materialization paths to every subscribed WS connection
and the journal. Fold the tag+ref pair into a {kind:'file'} part, same
as steered.
…ndow Two review fixes around prompt media intake: - The compressed re-save was released right after intake (and carried a 1h expiry) while every client read model projects its file id, leaving historical compressed images unfetchable. Keep the re-save as an ordinary upload; roll it back only when preparation or submission fails before the engine takes the prompt. The engine's PromptInput.release hook loses its only producer and is removed. - A prompt aborted while its steer awaited the loop's step assignment was flipped back to 'steered' and its content could still materialize into a later turn. Re-check the reservations after the assignment await and abort the undispatched request when the check fails.
# Conflicts: # apps/kimi-code/src/tui/kimi-tui.ts
A successful image inline depends only on the immutable upload bytes, so it is memoized per file id (size-bounded) in media.resolved and reused across steps, retries, and media-recovery reprojections instead of re-reading and re-encoding on every request. Degrade forms are never memoized since they depend on the message's tag pairing. Also make the never-empty message placeholder kind-aware (video vs image).
Edges (TUI, kap-server REST) now submit bare kimi-file references and the engine intake materializes the bytes, synthesizes the paired media path tag, and falls back to the shared cache dir when the session store is unavailable, replacing per-edge pair construction and duplicate materialization copies. Thread the prompt id from submission through to turn.started (REST prompt_id, WS event, SDK prompt option) so the TUI binds staged-media leases to turns exactly; the origin heuristic stays as fallback and ambiguous claims now surface a staging_lease_invariant telemetry warning. Also lands the pending resendable-extraction fix for cache-hint resubmits after a session switch.
Fold duplicate pinning of the same media tag+ref rules into shared helpers and it.each tables, and drop assertions that restate behavior already covered at another layer: - drop the kimiFileUrl alias describe (mediaRef.test.ts covers the aliased functions with more cases) - drop pairMediaPathTagRefs describe in favor of the parity fixtures - merge the identical prompt.steered/prompt.queued broadcast tests - parameterize the resolver degradation matrix and prompt intake fixtures (enqueueMedia/gatedImage/expectMediaPair helpers) - drop REST-level context-memory pairing assertions (engine-level intake tests pin the same shapes); keep the caption->system-reminder assertion, the only cover of extractCompressionCaptions - drop the turn-finish-during-intake steer-cancel vector and the switch-session release driver test (unit-level lease tests remain) Net -762 lines; 645 tests green across agent-core-v2, kap-server, transcript, node-sdk, klient, and the TUI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
No linked issue — the problem is explained below. Opening as a draft for early discussion per CONTRIBUTING.
Problem
Pasted/uploaded images are inlined into prompts as base64 data URLs with no on-disk file. As a result the model cannot re-read an image after the context projector drops old image parts (the "re-read the file" hint is a dead end for pasted images), cannot process it with tools (Bash/Python), and cannot feed it to file-based workflows. Video already has engine-native file references (
kimi-file://+ a resolver); images don't.What changed
Images now follow the same engine-native reference model as video, unified under one media-ref abstraction:
agent-core-v2'smediadomain;packages/kosongitself is untouched): newmediaRef.ts—kimi-file://URL helpers,MediaKindclassification, MIME/extension tables, and<image|video|audio path>tag build/match helpers, with a transcript-side contract copy atpackages/transcript/src/contract/mediaRef.ts.kimiFileUrl.tsand the file-type suffix table become aliases/re-exports.AgentVideoResolverService→AgentMediaResolverService(old names kept as deprecated aliases). Image strategy is inline-only: references resolve to base64 data URLs at LLM request time; when the referenced file is gone, the part is dropped if an adjacent<image path>tag already conveys the path (the model can re-read it with ReadMediaFile), otherwise an[image omitted]notice is emitted. Video semantics are byte-identical.[caption?, <image path> text, image_url kimi-file ref]; compression-altered bytes are saved as a new upload. Transcript projection emits{kind:'file'}without leaking internal URLs. The generic-file flow, theATTACHED_FILE_NOTICEwording, and the REST inline-base64 path are untouched.global.filessave/get/deleteover both memory and ipc transports (base64 wire format).harness.uploadFile()plus re-exported daemon file-reference helpers (changeset:@moonshot-ai/kimi-code-sdkminor).Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update. (behind the experimental v2 engine; no user-facing doc surface yet)