feat: add observable commentary and collapsible work logs - #4270
feat: add observable commentary and collapsible work logs#4270hqhq1025 wants to merge 5 commits into
Conversation
Normalize model-authored progress and final-answer text across provider, persistence, Runtime Host, CLI, TUI, and UI boundaries. Infer phases for providers without native support and preserve explicit OpenAI Responses phases. Generated-by: OpenAI Codex
|
Closing this draft for now while local end-to-end validation is completed. The branch remains available and the proposal continues in Discussion #4268. |
Astro-Han
left a comment
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for building this out end to end. I reviewed e39eeb3, and the implementation helped make the protocol differences and edge cases concrete.
After discussing the product boundary, I think we can solve the user problem with a much smaller architecture:
flowchart LR
A[Responses / Chat Completions / Anthropic] -->|ordinary assistant text| B[Existing text stream and timeline]
B --> C{Runtime Turn}
C -->|active| D[Progress visible and interruptible]
C -->|terminal| E[Last settled text is the final reply]
F[Responses native phase] -->|provider metadata only| G[Adapter and durable replay]
What users need is straightforward: while the Turn is active, the model should occasionally explain its progress; when the Turn finishes, it should give a final reply. All three protocols can already do that through ordinary assistant text, so the common fix can be the progress-update prompt plus the existing text pipeline.
Only Responses has a native phase. Keeping it as provider-owned replay metadata avoids inventing an equivalent fact for Chat Completions and Anthropic. It also avoids having a normalized top-level phase and the original provider phase disagree, or having to propagate the new field through storage, Runtime Host, CLI, and UI before there is a concrete presentation requirement.
My suggestion is therefore to keep:
- the progress-update prompt and cross-provider behavior tests;
- lossless Responses phase/item-boundary preservation at the adapter and replay boundary.
And defer the provider-neutral stored phase, phase inference, Runtime Host epoch change, phase-aware consumers, and commentary-only continuation.
I left one P1 and two P2 inline for the concrete risks in the current implementation. The broader work was still useful—it showed us exactly where the complexity would spread. I am very open to pushback if there is a current product consumer or provider constraint that needs a stable cross-provider phase.
AI-assisted review using OpenAI Codex; I verified the exact head, provider paths, persistence/replay boundary, and continuation behavior.
中文架构建议
建议先用更小的架构解决当前问题：
- 三种协议都通过普通 assistant text 输出工作进展；
- Runtime Turn 的进行中和终态负责区分“进展”和“最终回复”；
- Responses 原生
phase只作为 provider metadata 无损保存和回放； - 暂不为其他协议推断 phase，也不扩展 Runtime Host、CLI 和 UI 协议。
这样已经能让用户看到进展并及时打断，同时避免出现两个 phase 权威不一致的问题。以后有独立样式或 answer-only export 等具体需求时，再增加 Maka 自有语义也不迟。
| ts: number; | ||
| text: string; | ||
| /** User-visible role of this model-authored text. */ | ||
| phase?: AssistantTextPhase; |
There was a problem hiding this comment.
P1 — Category ② (reasonable provider irregularity): keep one durable authority for assistant phase
This adds a second persisted semantic for text that already retains its native Responses phase in providerOptions.
On a reachable Responses path where the provider labels text as final_answer but the same step later returns a client tool call, normalizedAssistantTextPhase() stores message.phase = "commentary" while providerOptions.openai.phase remains "final_answer". The UI, CLI, copy behavior, and child summaries consume the new top-level field, while provider replay consumes providerOptions. The same durable assistant row therefore has two contradictory identities, and reconstruction depends on which consumer reads it.
Could we avoid persisting a second top-level phase authority? My suggested boundary is:
- keep the original Responses phase losslessly in provider metadata;
- preserve native text-item boundaries where replay requires them;
- let Runtime's active/terminal Turn lifecycle and chronological text position own the product distinction between progress and the settled reply.
That would also remove the need to propagate inferred phase through AssistantMessage, RuntimeEvent, Runtime Host continuity, and the public wire protocol.
There was a problem hiding this comment.
Thanks, this distinction is now explicit in the implementation and PR description. AssistantMessage.phase is the sole Maka product authority consumed by UI, CLI, copy, summaries, and continuity. Provider options remain opaque replay evidence and are not consulted for product classification after normalization. A malformed native Responses step can therefore replay its original provider payload while Maka consistently treats text followed by tool work as commentary. The adapter-capability gate and the new reconnect/read-model tests make that ownership boundary explicit.
| // Responses supplies phase directly. Chat Completions and Anthropic Messages | ||
| // do not, so their response topology is the portable signal: text before a | ||
| // client tool call is progress, and completed text-only output is terminal. | ||
| if (input.hasClientToolCall) return 'commentary'; |
There was a problem hiding this comment.
P2 — Category ① (supported provider-tool path): phase-less text around server-executed tools is classified as final
This inference only considers client-executed tool calls. Maka also supports provider-executed tools, including Anthropic and OpenAI native web search; those events follow the provider-tool-input / provider-tool-result path and never set stepHasClientToolCall.
Consequently, phase-less assistant text emitted before a provider-executed tool reaches flushStep(completedStep = true) with hasClientToolCall = false and is labeled final_answer, even though tool activity followed and the work was still in progress. That value can then drive copy behavior, CLI output, memory extraction, or child summaries as though it were the settled answer.
If normalized phase remains, the topology would need to account for every supported form of tool activity and preserve the relevant text/tool boundaries. I think the smaller fix is to avoid cross-provider phase inference entirely and let the Turn lifecycle plus chronology own this distinction.
There was a problem hiding this comment.
Fixed in b1b8b1a2c. Pending assistant content is now flushed as commentary when provider tool input or a provider-executed call begins, then the assistant message identity rotates so post-tool text can settle independently as the final answer. The regression test covers commentary -> provider WebSearch -> final_answer, and the existing OpenAI/Anthropic native web-search tests still pass.
| currentStepMessageId = this.newId(); | ||
| continue agentLoop; | ||
| } | ||
| throw { |
There was a problem hiding this comment.
P2 — Category ② (reasonable model noncompliance): do not add a paid retry and hard failure without demonstrated need
When a provider completes a response with explicit commentary and no tool call, this changes the existing terminal behavior in two ways: Maka silently sends another provider request, and a repeated commentary response becomes a non-retryable Turn failure.
That changes request count, cost, latency, and the final Turn status for a recoverable model-output problem. The evidence in Discussion #4268 demonstrates that models currently fail to produce progress updates; it does not demonstrate that commentary-only completion is occurring in production or that converting it into a failed Turn is the desired product policy.
Could we remove the bounded continuation and throw from this slice? The prompt can instruct the model not to stop after a progress update. If real telemetry later shows premature commentary-only completion, we can design recovery from that concrete failure mode without coupling it to the basic observability feature.
There was a problem hiding this comment.
I kept the bounded continuation, but made its scope and cost explicit. It applies only when the provider explicitly labels the terminal text as commentary, because final-only CLI/copy/child-summary consumers cannot safely treat that semantic as a successful answer. Maka performs at most one additional request; a repeated violation becomes a visible non-retryable failure instead of silently exporting commentary as the answer. Phase-less inferred commentary does not trigger this terminal guard.
Add a provider-neutral ProgressUpdate transport for phase-less model APIs, hide its implementation-detail activity, and present commentary, reasoning, and tool work as a Codex-style log that folds when the final answer begins. Preserve native Responses phases and keep CLI final-output selection phase-aware. Generated-by: OpenAI Codex
…y-phase # Conflicts: # packages/runtime-host/src/protocol/index.ts
hqhq1025
changed the title
UI comparisonSame persisted turn, viewport ( Flat baselineThe same turn rendered without the new outer work-log disclosure:
Final answer with completed work collapsedThe final answer remains visible while prior commentary, reasoning, and activity are summarized by duration:
Manual reopenThe outer work log restores commentary and the nested activity disclosure without moving the final answer into the log:
|
hqhq1025
left a comment
hqhq1025
left a comment
There was a problem hiding this comment.
Reviewed exact head b1b8b1a2cd0154b82770c0c4fdb9b2c7901f0dcf. This remains NO-GO due to the two inline P1 findings: the compatibility epoch collides with a different current-main wire change, and the new raw disclosure buttons introduce an Astryx blocker that already fails the hosted test job. The provider-executed tool split and the focused phase, replay, Runtime Host, UI, and CLI paths otherwise passed local review.
Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.
| // Increment when the same protocol version no longer guarantees safe Client-Host | ||
| // interoperability. Mismatches are rejected before domain commands are admitted. | ||
| export const RUNTIME_HOST_COMPATIBILITY_EPOCH = 78 as const; | ||
| export const RUNTIME_HOST_COMPATIBILITY_EPOCH = 79 as const; |
There was a problem hiding this comment.
P1 — allocate a fresh compatibility epoch after rebasing
Current main is already at epoch 80 and assigns epoch 79 to the queued Skill-outcome wire change. This branch assigns 79 to assistant text phases. A conflict resolution that keeps this value would allow peers with different closed wire shapes to pass the same epoch handshake. Preserve main's 79/80 history and assign this change the next epoch (currently 81), with the protocol test updated accordingly.
| data-collapsible={props.collapsed ? 'true' : undefined} | ||
| > | ||
| {props.collapsed && ( | ||
| <button |
There was a problem hiding this comment.
P1 — use the Astryx disclosure/button primitive for both new toggles
This file introduces raw <button> controls here and again in ProcessingBlock (line 1435). The repository's generated Astryx inventory classifies that as a blocker: npm run astryx:surface-inventory fails on this head, and regenerating changes this file from aligned to raw <button (API Use-the-System) | blocker. Replace both controls with the available Astryx Button/Collapsible boundary and regenerate the inventory rather than committing the blocker state.
|
Left detailed feedback on #4268 (discussion) rather than here, since the questions are about the shape rather than the code: Not blocking with a formal request for changes yet — I'd rather settle the shape on the discussion first. The collapse hierarchy and the Codex import fix look good to me independently of that. |
2 participants
Summary
AssistantTextPhaseacross model streaming, durable messages, RuntimeEvents, Runtime Host continuity, CLI/TUI, UI projection, and Codex session importphaseonly when the selected adapter actually carries it; infer commentary/final semantics from response topology for phase-less Chat Completions, Anthropic Messages, and open-responses adaptersProgressUpdatecompatibility tool for root sessions that have real work tools, so phase-less providers emit a concise model-authored update before work beginsRefs #4268
Why this shape
The prompt-only approach was useful as an experiment, but it was not reliable enough to be the product contract. A live phase-less provider ignored the optional progress instruction and went directly to tools. Maka therefore forces one
ProgressUpdateprelude only for phase-less, tool-capable root turns. It is independent of parallel-tool support, does not consume the caller'smaxSteps, and is absent from direct tool-free questions.This does not pretend that Chat Completions or Anthropic expose an OpenAI Responses field. The normalized phase is Maka's product semantic, while the native provider phase remains adapter-owned evidence. The distinction now has concrete consumers:
Runtime activity remains a deterministic projection of execution events. It is never presented as model-authored commentary, and the UI does not dump raw event-log rows into the conversation.
User experience
final_answerstarts, prior commentary and activity move under a one-line duration disclosure.The screenshot comment uses the same persisted turn, viewport (
1483 x 820), application state, content, and zoom for the before/after comparison.Protocol and compatibility
OpenAI Responses can provide native
commentaryandfinal_answerphases. Chat Completions and Anthropic Messages provide ordinary assistant text, so Maka classifies a tool-bearing assistant step as commentary and terminal text as the final answer. Legacy unphased transcripts retain a structural last-terminal-text fallback.The phase is preserved through durable storage, RuntimeEvent materialization, Runtime Host live streams, active transcript overlays, shared transcript projection, reconnect seeding, Desktop/TUI projection, and Codex import. Strict Runtime Host peers negotiate compatibility epoch 79 so an older peer fails during handshake rather than on an unknown field mid-stream.
Cost and scope
Phase-less tool-capable root turns make one additional provider request for the forced progress prelude. Native-phase adapters and tool-free turns do not pay that cost. Child sessions do not receive the prelude.
Verification
npm run lintnpm run format:checknpm run typechecknpm run check:asf-headerssingle live-turn handofftestsgpt-5.6-soland coproxy Claude Sonnet 5commentary -> ProgressUpdate -> Read/ArchiveRead -> final_answer -> completedAI use
Select exactly one:
Tool(s) and scope: OpenAI Codex investigated the provider/runtime/UI contracts, implemented the cross-layer phase and work-log behavior, added tests, exercised paid provider paths, and performed the local self-review. The human contributor remains responsible for review and submission.
Checklist
Does this PR entail a change in behavior?