diff --git a/README.md b/README.md index b7157ac..04b00fe 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,8 @@ Inspired by [Scott Hanselman's "AI without the BS, for humans" keynote at NDC Lo - **Taught once, then out of the way.** Three coach marks run in order for a first-time visitor: the color rule on the first settled reply, the reply tabs on the newest unlocked one, and the cost card once two replies exist (that one opens the card's **Details** for you). Steps 1 and 2 advance on the gesture they teach — hovering a word, picking a tab — as well as on **Got it**, so following the instruction is never punished with an extra click. Progress is a single number in `localStorage`, so they do not come back. After that the same three sentences stay reachable behind `?` buttons on the legend, the tab strip, and the conversation-cost line; all three read from `lib/coachCopy.js`, so a mark and its `?` cannot drift apart. - **What else was considered.** Hover or tap a word for the candidate list, headed **What it considered**. It opens on **Of all words**: the model’s real probabilities across the whole vocabulary, which do *not* add up to 100% — the honest number first. **What-if: only these** re-scales just the shown candidates to add up to 100% at the current temperature, which is a different quantity and says so. If the sampled token was outside the top 5, it still gets its own row (“landed — not in the top 5”) with a real percentage instead of `0.00%`; under **Of all words**, that is its exact model probability. - **Temperature, live.** The candidate set is frozen when the card opens, so moving the temperature slider never makes rows appear or vanish — only the odds move. Adjusting temperature does not dismiss a pinned card, because watching the odds shift is the lesson. At `0` the top candidate takes 100% and everything else goes to zero, which is winner-take-all sampling made visible (the card rounds those zeros to `<0.001%`). The full panel behind the header's **Controls** button is grouped into **Sampling** (temperature, top-p, presence penalty, a **Make it repeatable** determinism switch), **Memory** (**Forget older turns** and **Exchanges replayed**), **Delivery** (**Stream the reply**), and **Tools** (**Let it call a weather tool**). The button always carries a `temp N.N` chip, plus one chip for every switch that has moved off its default — `streaming off`, `memory none` or `memory last N`, `tool on`, `repeatable` — so no switch can be on without the header saying so. The two secondary sliders, top-p and presence penalty, earn no chip even though every request carries them, so the panel is the only place that reports where they sit. -- **Your text is tokens too.** The composer tokenizes what you type with `o200k_base` and the user bubble shows the pieces as alternating tints, with an `≈ N tokens` count. Send **strawberry** and watch it arrive as three pieces, not ten letters. -- **Three full replies, and where they fork.** Each turn requests `n=3` completions. A strip reading “3 replies were written. You are reading reply” fronts tabs **1 / 2 / 3**, each with a confidence dot, and an unlocked strip carries a `?` that repeats the coach-mark sentence; older turns lock after the next user message, and a padlock explains why. A ring marks the first token where the three replies diverge — everything before it is identical, because the same prompt and the same weights produced the same tokens until the dice landed differently. Each tab also reports perplexity (“picking from ~N plausible words”). +- **Your text is tokens too.** The composer tokenizes what you type with `o200k_base` and the user bubble shows the pieces as alternating tints, with an `≈ N tokens` count that becomes an addition once the reply's usage lands: `N this message + M system and wrappers = P input tokens` on the first send, and `R from earlier turns + L last reply + N this message + W wrappers = P input tokens` after that. The terms sum to the billed input; the remainder is what the tokenizer cannot name. When tools first ride along, that remainder's label names the schema. Send **strawberry** and watch it arrive as three pieces, not ten letters. +- **Three full replies, and where they fork.** Each turn requests `n=3` completions. A strip reading “3 replies were written. You are reading reply” fronts tabs **1 / 2 / 3**, each with a confidence dot, and an unlocked strip carries a `?` that repeats the coach-mark sentence; older turns lock after the next user message, and a padlock explains why. A ring marks the first token where the three replies diverge — everything before it is identical, because the same prompt and the same weights produced the same tokens until the dice landed differently. Each tab also reports its length and perplexity (“14 tokens · picking from ~N plausible words”), and a muted `N tokens` sits after the last word of the reply you are reading. - **Conversation cost, in tokens first.** The API has no memory. Every turn resends the whole prompt, so input tokens climb as a staircase — one stacked bar per request, split into replayed, cached, and new. A card at the foot of the transcript, **What each request carried**, leads with the two numbers that actually teach the lesson: *Sent this turn* (`143 in · 13 out`, or `270 + 341 in · 49 out · 2 requests` on a tool turn) and *Conversation so far* in tokens. Dollars are the footnote, not the headline — open **Details** for the staircase, the running spend, the literal JSON array that was sent, and a **How is this priced?** disclosure holding the rate card (for the default model, $0.15 / 1M in, $0.60 / 1M out, $0.075 / 1M cached in) and this turn's input / cached-input / output split. Each message also carries its own usage line, now tokens only (`N in · M out`), with the dollar figure moved into its expandable breakdown as `— this turn at list price`. - **Fractions of a cent, spelled out.** A turn on `gpt-4o-mini` costs far less than a cent, and `$0.00` teaches nothing. `formatUsd` in `lib/openaiRates.js` prints `$0.02` at or above a cent, `$0.01` for anything from about two-thirds of a cent up (that *is* “about a cent”), `≈ 1/167 of a cent` below that, and `less than 1/10,000 of a cent` at the floor. Beside the conversation total, `formatScale` multiplies the last turn by a million to give the number a size a person can hold: `a million chats like this ≈ $60.00`. - **No memory, made visible.** The model has no memory of its own; the app replays the transcript every request. Turn on **Forget older turns** and the request stops carrying the top of the chat — a line appears in the transcript, the turns above it dim, and the model can no longer answer a question about a fact you seeded before the line. The transcript and your saved conversation keep everything; only the request shrinks. The system prompt never falls off, because the server adds it every time. The empty screen offers the demo as a path rather than a puzzle: a **Give it a fact to remember** chip seeds “My name is Ada. Remember it.”, and once that reply settles a **Now make it forget** chip appears, flips **Exchanges replayed** to `0`, and asks “What is my name?” in one click. The weather question has the same shape: after any reply to a prompt that mentions weather, with the tool off, a **Now give it the tool** chip turns the tool on and re-sends the same question, so the two replies sit side by side. diff --git a/components/ChatInterface.js b/components/ChatInterface.js index 2c1c2b2..3323490 100644 --- a/components/ChatInterface.js +++ b/components/ChatInterface.js @@ -17,7 +17,7 @@ import { pruneForStorage } from '../lib/persistence'; import { abortedFields, abortedTurn } from '../lib/abortedTurn'; import { buildOutboundMessages, KEEP_ALL, KEEP_TURNS_DEFAULT } from '../lib/contextWindow'; import { knowledgeCutoff } from '../lib/modelFacts'; -import { formatTokenSummary } from '../lib/usage'; +import { formatTokenSummary, offeredTools, selectedReplyTokens } from '../lib/usage'; import { needsCutoffNote, mentionsWeather } from '../lib/cutoffRelevance'; import { COACH_TEXT_COLOR, COACH_TEXT_TABS, COACH_TEXT_COST } from '../lib/coachCopy'; @@ -933,6 +933,12 @@ export default function ChatInterface() { const cutoffPrompt = [...messages.slice(0, index)] .reverse() .find((item) => item.role === 'user') ?? null; + const followingAssistant = message.role === 'user' + ? messages.slice(index + 1).find((item) => item.role === 'assistant') + : null; + const priorAssistant = message.role === 'user' + ? [...messages.slice(0, index)].reverse().find((item) => item.role === 'assistant' && item.usage?.prompt_tokens != null) + : null; const node = ( item.role === 'user')} tokenizer={tokenizer} forgotten={forgetting.truncated && index < forgetting.cutoffIndex} diff --git a/components/Message.js b/components/Message.js index 3d149ef..1715a66 100644 --- a/components/Message.js +++ b/components/Message.js @@ -4,7 +4,7 @@ import CoachMark from './CoachMark'; import { tokenizeForDisplay, isPartialChunk } from '../lib/tokenizer'; import { sampledLogprob, findForkIndex, completionStats, formatPerplexity, confidenceColor, confidenceParts, confidenceBand } from '../lib/completionStats'; import { rateFor, turnCost, formatUsd } from '../lib/openaiRates'; -import { formatTokenSummary } from '../lib/usage'; +import { formatTokenSummary, formatUserTokenLine, offeredTools } from '../lib/usage'; import { knowledgeCutoff } from '../lib/modelFacts'; import { mentionsWeather } from '../lib/cutoffRelevance'; import { COACH_TEXT_TABS } from '../lib/coachCopy'; @@ -61,7 +61,7 @@ const tokenHeatStyle = (tokenData) => { return { '--conf-rgb': parts.rgb, '--conf-rgb-dark': parts.rgbDark, '--conf-a': parts.alpha, '--conf-on': 1 }; }; -function Message({ message, onSelect, messageIndex, coach = null, onCoachAdvance, sessionBilled, replayedIn, addedIn, tabsLocked = false, tokenizer, forgotten = false, showCutoffDetail = false, cutoffPrompt = null }) { +function Message({ message, onSelect, messageIndex, coach = null, onCoachAdvance, sessionBilled, replayedIn, addedIn, tabsLocked = false, tokenizer, forgotten = false, showCutoffDetail = false, cutoffPrompt = null, promptIn = null, toolsOffered = false, replayedPromptIn = null, lastReplyTokens = null, previousToolsOffered = false }) { const { role, completions, activeIndex = 0, content } = message; const isStreaming = !!message.isStreaming; const [hoveredToken, setHoveredToken] = useState(null); @@ -323,6 +323,9 @@ function Message({ message, onSelect, messageIndex, coach = null, onCoachAdvance ); })} + {tokenProbabilities.length > 0 && ( + {tokenProbabilities.length} tokens + )} ); }; @@ -348,6 +351,7 @@ function Message({ message, onSelect, messageIndex, coach = null, onCoachAdvance const stats = tabStats[index]; const parts = [ `Response ${index + 1}`, + stats?.tokenCount != null && `${stats.tokenCount} tokens`, stats && formatPerplexity(stats.perplexity), ].filter(Boolean); return ( @@ -514,14 +518,26 @@ function Message({ message, onSelect, messageIndex, coach = null, onCoachAdvance )} - {role === 'user' && userChunks && ( - - ≈ {userChunks.length} tokens - - )} + {role === 'user' && userChunks && (() => { + const line = formatUserTokenLine({ + messageTokens: userChunks.length, + promptTokens: promptIn, + replayedTokens: replayedPromptIn, + lastReplyTokens, + toolsOffered, + previousToolsOffered, + }); + const title = !Number.isFinite(promptIn) + ? 'Counted with the same tokenizer the model family uses. The reply\'s \'new\' number runs a little higher — the chat wrapper rides along with every turn.' + : toolsOffered + ? 'Each term is billed tokens except this message and last reply, which are the tokenizer cut of those bodies. The remainder is wrappers — and the tool schema, when it first rides along. They add up to the input tokens the API billed.' + : 'Each term is billed tokens except this message and last reply, which are the tokenizer cut of those bodies. The remainder is the system prompt and the chat wrapper. They add up to the input tokens the API billed.'; + return ( + + {line} + + ); + })()} {message.usage?.prompt_tokens != null && (() => { const summary = formatTokenSummary(message.usage); return ( @@ -545,6 +561,9 @@ function Message({ message, onSelect, messageIndex, coach = null, onCoachAdvance {usageOpen && message.usage?.prompt_tokens != null && (
{rounds ? `${message.usage.prompt_tokens} in — everything sent this turn, across two requests` : `${message.usage.prompt_tokens} in — everything sent this request`} + {offeredTools(message) && ( + the weather tool schema rode with this request — counted in the prompt + )} {rounds && rounds.length > 1 && rounds.map((r, i) => ( {r.prompt_tokens} in · {r.completion_tokens} out — {i === 0 ? 'first request, the one that ended in a tool call' : 'next request, the same prompt plus the tool call and its result'} diff --git a/docs/inside-chatprob.md b/docs/inside-chatprob.md index 6baf115..fdc9440 100644 --- a/docs/inside-chatprob.md +++ b/docs/inside-chatprob.md @@ -52,9 +52,9 @@ Every piece of text that touches the model goes through this cut: your message, ### Try it — step 1 -Type **strawberry** into the composer and send it. Nothing about the text changes while you type — the composer is a plain text box. The tokenizer does its work on the user bubble that appears once you send: the app runs the same `o200k_base` cut on your message and paints the pieces in alternating tints, with an `≈ N tokens` count underneath. *strawberry* arrives as three pieces, not ten letters. That is why "how many r's are in strawberry" is a hard question for a model: it has to reason about letters it never received, inside chunks it cannot see into. +Type **strawberry** into the composer and send it. Nothing about the text changes while you type — the composer is a plain text box. The tokenizer does its work on the user bubble that appears once you send: the app runs the same `o200k_base` cut on your message and paints the pieces in alternating tints, with an `≈ N tokens` count underneath. Once the reply lands that line becomes an addition that sums to the billed input — `N this message + M system and wrappers = P input tokens`. Later sends insert last turn's input as `from earlier turns` and last turn's selected reply as `last reply`. *strawberry* arrives as three pieces, not ten letters. That is why "how many r's are in strawberry" is a hard question for a model: it has to reason about letters it never received, inside chunks it cannot see into. -The count's tooltip carries an honesty note: the reply's own "new tokens" number in chapter 7 runs a little higher, because the API also counts the chat wrapper around each message. +The count's tooltip carries an honesty note: the remainder is what the API bills but the tokenizer cannot name — the system prompt and the chat wrapper around each message. A `·` in a user bubble is a token that decoded to nothing printable on its own — a fragment of a multi-byte character. @@ -312,7 +312,7 @@ The model is paid by the token, in both directions, and chapter 6 just showed yo Look at the card's header before opening **Details**. **Sent this turn** — something like `143 in · 13 out`. **Conversation so far** — a running token total. These are the two numbers that teach the lesson, and they are tokens, not money, on purpose. **[Input tokens](glossary.md#input-tokens)** (the API calls them prompt tokens) are everything in the request: system prompt, every replayed message, the chat wrapper around each one, your new message. **[Output tokens](glossary.md#output-tokens)** (completion tokens) are what the model wrote — and because each request draws three replies, the `out` number counts all three, not just the tab you are reading. Together they are the request's **[usage](glossary.md#usage)**, reported by the API at the end of every response. -The same summary sits under each reply as `N in · M out`, with a `▾` that opens the itemized version: *everything sent this request*, *replayed — last turn's prompt, sent again*, *new — last reply plus your latest message*, *out this tab*, *total out this turn — all samples*, and the dollar figure for the turn at list price. +The same billed input sits on the user bubble as an addition that sums to it — `N this message + M system and wrappers = P input tokens` on the first send, and `R from earlier turns + L last reply + N this message + W wrappers = P input tokens` after that — so you do not have to open the card to see how the request grew. Under each reply, `N in · M out` is the receipt for the whole turn, with a `▾` that opens the itemized version: *everything sent this request*, *replayed — last turn's prompt, sent again*, *new — last reply plus your latest message*, *out this tab*, *total out this turn — all samples*, and the dollar figure for the turn at list price. After the last word of the reply itself, a muted `N tokens` is the length of the tab you are reading — switch tabs and that number changes. ### The staircase @@ -338,7 +338,7 @@ With the cost card open: read the two numbers; find the pale segment growing fro ### In this repo -`buildUsage` and `formatTokenSummary` in `lib/usage.js` build the per-turn `usage` object from the API's `prompt_tokens`, `completion_tokens` and `prompt_tokens_details.cached_tokens`, and print the `N in · M out` line. `rateFor`, `turnCost`, `formatUsd` and `formatScale` in `lib/openaiRates.js` hold the rate card and the fraction-of-a-cent and million-chat formatters. `components/PromptStaircase.js` draws one bar per request and splits it against the previous request's prompt; `CostFooter` and `ConversationExplainer` in `components/ConversationExplainer.js` are the price lines and the teacher copy. +`buildUsage` and `formatTokenSummary` in `lib/usage.js` build the per-turn `usage` object from the API's `prompt_tokens`, `completion_tokens` and `prompt_tokens_details.cached_tokens`, and print the `N in · M out` line. `formatUserTokenLine` is the user-bubble pair; `usage.tools` is set only when the server attached the weather schema. `rateFor`, `turnCost`, `formatUsd` and `formatScale` in `lib/openaiRates.js` hold the rate card and the fraction-of-a-cent and million-chat formatters. `components/PromptStaircase.js` draws one bar per request and splits it against the previous request's prompt; `CostFooter` and `ConversationExplainer` in `components/ConversationExplainer.js` are the price lines and the teacher copy. ## 8. What it cannot know @@ -388,7 +388,7 @@ The first card's badge says **the model asked for a tool**. Beside it, `get_weat } ``` -The request went out as before — system prompt, conversation, your question — plus a `tools` array beside the messages holding that schema. Instead of a sentence, the model wrote a **[tool call](glossary.md#tool-call)**: a structured message saying *call this function with these arguments*. It emitted the function name and the JSON argument token by token, exactly the way it writes words, sampled from a distribution like anything else. It did not run anything. It cannot. It produced a request. +The request went out as before — system prompt, conversation, your question — plus a `tools` array beside the messages holding that schema. The user bubble for this send names the schema in the remainder of the addition (`wrappers and the tool schema`); the first Denver send, tools off, does not. The schema is in that billed total even if a sample answers in prose and never calls the function. Instead of a sentence, the model wrote a **[tool call](glossary.md#tool-call)**: a structured message saying *call this function with these arguments*. It emitted the function name and the JSON argument token by token, exactly the way it writes words, sampled from a distribution like anything else. It did not run anything. It cannot. It produced a request. The card's note says the rest, and it is worth quoting because it includes a deliberate gap: *The API returns no probabilities for these tokens, so there is nothing to shade here.* The argument JSON is sampled text, but the API does not expose logprobs over tool-call arguments, so the card prints it plain rather than pretending to a heatmap it does not have. That absence was measured, not assumed. @@ -429,7 +429,7 @@ Ask for two cities in one question and the model may emit two tool calls in a si ### In this repo -`WEATHER_TOOLS` and `parseWeatherArguments` in `lib/weatherTool.js` are the schema the model is shown — the same object the panel renders — and the argument check. `getWeather` and `normalizeLocation` in `lib/weather.js` make the call; the key is read there and nowhere else. In `pages/api/chat.js`, round one sends `tools`, the route picks the first choice with `tool_calls`, `MAX_TOOL_CALLS` caps execution, and round two appends the `assistant` tool-call message and one `role: "tool"` message per call before sending `tool_choice: "none"`. `buildUsage` in `lib/usage.js` sums both rounds and keeps them under `usage.rounds`; the client sends `tools: true` and skips streaming in `components/ChatInterface.js`; the cards are in `components/Message.js`. `scripts/tool-spike.mjs` is the live check that tools, three replies and logprobs work together, and that logprobs are absent over tool-call arguments. +`WEATHER_TOOLS` and `parseWeatherArguments` in `lib/weatherTool.js` are the schema the model is shown — the same object the panel renders — and the argument check. `getWeather` and `normalizeLocation` in `lib/weather.js` make the call; the key is read there and nowhere else. In `pages/api/chat.js`, round one sends `tools`, the route picks the first choice with `tool_calls`, `MAX_TOOL_CALLS` caps execution, and round two appends the `assistant` tool-call message and one `role: "tool"` message per call before sending `tool_choice: "none"`. `buildUsage` in `lib/usage.js` sums both rounds and keeps them under `usage.rounds`, and sets `usage.tools` so the user bubble can still name the schema after `echoedTools` is dropped; the client sends `tools: true` and skips streaming in `components/ChatInterface.js`; the cards are in `components/Message.js`. `scripts/tool-spike.mjs` is the live check that tools, three replies and logprobs work together, and that logprobs are absent over tool-call arguments. ## 10. What the page keeps diff --git a/lib/usage.js b/lib/usage.js index 921f02e..6001c3a 100644 --- a/lib/usage.js +++ b/lib/usage.js @@ -13,8 +13,9 @@ function roundUsage(raw) { // One turn can cost more than one request. The totals are what you paid; // `rounds` is the itemised receipt, and only appears when there is more -// than one line on it. -function buildUsage(rawUsages, model, sampling) { +// than one line on it. `tools: true` is persisted only when the server +// actually attached the weather schema — never when the switch was off. +function buildUsage(rawUsages, model, sampling, offeredTools) { const rounds = rawUsages.map(roundUsage); return { prompt_tokens: sumNullable(rounds.map((r) => r.prompt_tokens)), @@ -23,6 +24,7 @@ function buildUsage(rawUsages, model, sampling) { model, sampling, ...(rounds.length > 1 ? { rounds } : {}), + ...(offeredTools === true ? { tools: true } : {}), }; } @@ -41,4 +43,106 @@ function formatTokenSummary(usage) { return parts.join(' · '); } -export { sumNullable, roundUsage, buildUsage, formatTokenSummary }; +// The selected tab's token count — the text that actually gets replayed. +function selectedReplyTokens(assistant) { + if (!assistant?.completions?.length) return null; + const active = assistant.completions[assistant.activeIndex || 0]; + const n = active?.tokenProbabilities?.length; + return Number.isFinite(n) && n > 0 ? n : null; +} + +function remainderLabel(kind, toolsInRemainder) { + if (kind === 'first') { + return toolsInRemainder ? 'system, wrappers, and the tool schema' : 'system and wrappers'; + } + return toolsInRemainder ? 'wrappers and the tool schema' : 'wrappers'; +} + +// Highest-grain billed remainder chain for a user bubble. +// First send: 5 this message + 45 system and wrappers = 50 input tokens +// Later send: 50 from earlier turns + 43 last reply + 5 this message + 8 wrappers = 106 input tokens +// Addends always sum to the billed prompt; if they cannot, we fall back to a coarser line. +function userPromptBreakdown({ + messageTokens, + promptTokens, + replayedTokens, + lastReplyTokens, + toolsOffered, + previousToolsOffered, +} = {}) { + if (!Number.isFinite(messageTokens) || !Number.isFinite(promptTokens)) return null; + const tools = toolsOffered === true; + const toolsNew = tools && previousToolsOffered !== true; + const replayed = Number.isFinite(replayedTokens) && replayedTokens > 0 && replayedTokens <= promptTokens + ? replayedTokens + : null; + + const addends = []; + if (replayed != null) { + addends.push({ tokens: replayed, label: 'from earlier turns' }); + const added = promptTokens - replayed; + const lastReply = Number.isFinite(lastReplyTokens) && lastReplyTokens > 0 ? lastReplyTokens : null; + if (lastReply != null) { + addends.push({ tokens: lastReply, label: 'last reply' }); + addends.push({ tokens: messageTokens, label: 'this message' }); + const wrap = added - lastReply - messageTokens; + if (wrap > 0) addends.push({ tokens: wrap, label: remainderLabel('later', toolsNew) }); + } else { + addends.push({ tokens: messageTokens, label: 'this message' }); + const rest = added - messageTokens; + if (rest > 0) addends.push({ tokens: rest, label: toolsNew ? 'last reply, wrappers, and the tool schema' : 'last reply and wrappers' }); + } + } else { + addends.push({ tokens: messageTokens, label: 'this message' }); + const rest = promptTokens - messageTokens; + if (rest > 0) addends.push({ tokens: rest, label: remainderLabel('first', tools) }); + } + + const sum = addends.reduce((total, part) => total + part.tokens, 0); + if (sum !== promptTokens) return null; + return { addends, promptTokens, tools, toolsNamedInRemainder: (replayed != null ? toolsNew : tools) && addends.some((part) => part.label.includes('tool schema')) }; +} + +// The one-line count under a user bubble. Accepts either the old +// (messageTokens, promptTokens, toolsOffered) positional form or a breakdown object. +function formatUserTokenLine(messageTokens, promptTokens, toolsOffered) { + const details = messageTokens != null && typeof messageTokens === 'object' + ? messageTokens + : { messageTokens, promptTokens, toolsOffered }; + if (!Number.isFinite(details.messageTokens)) return null; + if (!Number.isFinite(details.promptTokens)) return `≈ ${details.messageTokens} tokens`; + + const breakdown = userPromptBreakdown(details); + if (!breakdown) { + const prompt = `${details.promptTokens} input tokens${details.toolsOffered ? ', including the tool schema' : ''}`; + return `${details.messageTokens} this message · ${prompt}`; + } + const sum = breakdown.addends.map((part) => `${part.tokens} ${part.label}`).join(' + '); + const toolsClause = details.toolsOffered && !breakdown.toolsNamedInRemainder + ? ', including the tool schema' + : ''; + return `${sum} = ${breakdown.promptTokens} input tokens${toolsClause}`; +} + +// Durable signal: usage.tools survives echoedTools being dropped on the +// next send. The fallbacks cover in-flight turns and older transcripts +// saved before that flag existed — both only appear after a tools-on send. +function offeredTools(assistant) { + if (!assistant) return false; + if (assistant.usage?.tools === true) return true; + if (Array.isArray(assistant.echoedTools) && assistant.echoedTools.length > 0) return true; + if (assistant.toolCall) return true; + if (Array.isArray(assistant.toolCalls) && assistant.toolCalls.length > 0) return true; + return false; +} + +export { + sumNullable, + roundUsage, + buildUsage, + formatTokenSummary, + selectedReplyTokens, + userPromptBreakdown, + formatUserTokenLine, + offeredTools, +}; diff --git a/lib/usage.test.js b/lib/usage.test.js index 108ee6c..b7edffa 100644 --- a/lib/usage.test.js +++ b/lib/usage.test.js @@ -1,6 +1,6 @@ import test from 'node:test'; import assert from 'node:assert/strict'; -import { sumNullable, buildUsage, formatTokenSummary } from './usage.js'; +import { sumNullable, buildUsage, formatTokenSummary, formatUserTokenLine, userPromptBreakdown, selectedReplyTokens, offeredTools } from './usage.js'; test('buildUsage single-round output has exactly the expected keys, in order, and no rounds', () => { const usage = buildUsage( @@ -97,3 +97,148 @@ test('formatTokenSummary omits the out segment when completion_tokens is missing assert.equal(formatTokenSummary({ completion_tokens: 5 }), null); assert.equal(formatTokenSummary(null), null); }); + +test('buildUsage omits tools unless the fourth argument is true', () => { + const off = buildUsage( + [{ prompt_tokens: 10, completion_tokens: 2 }], + 'gpt-4o-mini', + { temperature: 1 } + ); + assert.equal('tools' in off, false); + + const on = buildUsage( + [{ prompt_tokens: 10, completion_tokens: 2 }], + 'gpt-4o-mini', + { temperature: 1 }, + true + ); + assert.equal(on.tools, true); + + const coerced = buildUsage( + [{ prompt_tokens: 10, completion_tokens: 2 }], + 'gpt-4o-mini', + { temperature: 1 }, + false + ); + assert.equal('tools' in coerced, false); +}); + +test('formatUserTokenLine is the in-flight count until prompt tokens arrive', () => { + assert.equal(formatUserTokenLine(6), '≈ 6 tokens'); + assert.equal(formatUserTokenLine(6, null), '≈ 6 tokens'); + assert.equal(formatUserTokenLine(NaN, 143), null); +}); + +test('formatUserTokenLine first send is this message plus the unitemised remainder', () => { + assert.equal( + formatUserTokenLine(5, 50), + '5 this message + 45 system and wrappers = 50 input tokens' + ); + assert.equal( + formatUserTokenLine(5, 50, true), + '5 this message + 45 system, wrappers, and the tool schema = 50 input tokens' + ); +}); + +test('formatUserTokenLine later send is the billed remainder chain', () => { + assert.equal( + formatUserTokenLine({ + messageTokens: 5, + promptTokens: 106, + replayedTokens: 50, + lastReplyTokens: 43, + }), + '50 from earlier turns + 43 last reply + 5 this message + 8 wrappers = 106 input tokens' + ); +}); + +test('formatUserTokenLine names the schema in the remainder when tools just turned on', () => { + assert.equal( + formatUserTokenLine({ + messageTokens: 5, + promptTokens: 186, + replayedTokens: 50, + lastReplyTokens: 43, + toolsOffered: true, + previousToolsOffered: false, + }), + '50 from earlier turns + 43 last reply + 5 this message + 88 wrappers and the tool schema = 186 input tokens' + ); +}); + +test('formatUserTokenLine names the schema on the total when it was already in history', () => { + assert.equal( + formatUserTokenLine({ + messageTokens: 5, + promptTokens: 186, + replayedTokens: 130, + lastReplyTokens: 43, + toolsOffered: true, + previousToolsOffered: true, + }), + '130 from earlier turns + 43 last reply + 5 this message + 8 wrappers = 186 input tokens, including the tool schema' + ); +}); + +test('userPromptBreakdown refuses a chain that does not sum to the billed prompt', () => { + assert.equal(userPromptBreakdown({ + messageTokens: 5, + promptTokens: 106, + replayedTokens: 50, + lastReplyTokens: 80, + }), null); + assert.equal( + formatUserTokenLine({ + messageTokens: 5, + promptTokens: 106, + replayedTokens: 50, + lastReplyTokens: 80, + }), + '5 this message · 106 input tokens' + ); +}); + +test('formatUserTokenLine without a last-reply count keeps last reply inside the new remainder', () => { + assert.equal( + formatUserTokenLine({ + messageTokens: 5, + promptTokens: 106, + replayedTokens: 50, + }), + '50 from earlier turns + 5 this message + 51 last reply and wrappers = 106 input tokens' + ); +}); + +test('userPromptBreakdown treats a replayed count larger than this prompt as a first send', () => { + const breakdown = userPromptBreakdown({ + messageTokens: 5, + promptTokens: 40, + replayedTokens: 50, + lastReplyTokens: 43, + }); + assert.deepEqual(breakdown.addends, [ + { tokens: 5, label: 'this message' }, + { tokens: 35, label: 'system and wrappers' }, + ]); +}); + +test('selectedReplyTokens reads the active tab length', () => { + assert.equal(selectedReplyTokens(null), null); + assert.equal(selectedReplyTokens({ + activeIndex: 1, + completions: [ + { tokenProbabilities: [1, 2] }, + { tokenProbabilities: [1, 2, 3, 4] }, + ], + }), 4); +}); + +test('offeredTools keys on usage.tools and falls back to echo or tool cards', () => { + assert.equal(offeredTools(null), false); + assert.equal(offeredTools({ usage: { prompt_tokens: 10 } }), false); + assert.equal(offeredTools({ usage: { tools: true } }), true); + assert.equal(offeredTools({ echoedTools: [{ type: 'function' }] }), true); + assert.equal(offeredTools({ echoedTools: [] }), false); + assert.equal(offeredTools({ toolCall: { name: 'get_weather' } }), true); + assert.equal(offeredTools({ toolCalls: [{ name: 'get_weather' }] }), true); +}); diff --git a/pages/api/chat.js b/pages/api/chat.js index 9315ba4..16a99ef 100644 --- a/pages/api/chat.js +++ b/pages/api/chat.js @@ -190,7 +190,7 @@ export default async function handler(req, res) { echoedMessages: sentMessages, echoedTools: wantsTools ? WEATHER_TOOLS : null, echoedToolChoice: null, - usage: buildUsage([response.usage], response.model || process.env.OPENAI_MODEL || 'gpt-4o-mini', samplingSnapshot), + usage: buildUsage([response.usage], response.model || process.env.OPENAI_MODEL || 'gpt-4o-mini', samplingSnapshot, wantsTools || undefined), }); } @@ -271,7 +271,7 @@ export default async function handler(req, res) { echoedMessages: round2Messages, echoedTools: WEATHER_TOOLS, echoedToolChoice: 'none', - usage: buildUsage([response.usage, round2.usage], round2.model || process.env.OPENAI_MODEL || 'gpt-4o-mini', samplingSnapshot), + usage: buildUsage([response.usage, round2.usage], round2.model || process.env.OPENAI_MODEL || 'gpt-4o-mini', samplingSnapshot, true), toolCalls: calls, toolResults: results, toolCall: calls[0], diff --git a/styles/panels.css b/styles/panels.css index 24e3ef0..6409aa8 100644 --- a/styles/panels.css +++ b/styles/panels.css @@ -7,11 +7,26 @@ .prompt-staircase-swatch.is-cached { background-color: var(--violet-3); } .prompt-staircase-swatch.is-replayed { background-color: var(--violet-2); box-shadow: inset 0 0 0 1px rgba(var(--violet-rgb), 0.25); } .prompt-staircase-swatch.is-new { background-color: var(--violet); } -.prompt-staircase-rows { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 3px; } -.prompt-staircase-row { display: grid; grid-template-columns: 2ch 1fr auto; align-items: center; gap: 8px; font-size: var(--fs-1); color: var(--muted); } +.prompt-staircase-rows { + list-style: none; + margin: 0; + padding: 0; + display: grid; + grid-template-columns: 2ch minmax(0, 1fr) auto; + column-gap: 8px; + row-gap: 3px; +} +.prompt-staircase-row { + display: grid; + grid-template-columns: subgrid; + grid-column: 1 / -1; + align-items: center; + font-size: var(--fs-1); + color: var(--muted); +} .prompt-staircase-turn { font-variant-numeric: tabular-nums; text-align: right; } .prompt-staircase-bar { display: flex; height: 12px; border-radius: var(--r-1); overflow: hidden; min-width: 2px; } -.prompt-staircase-seg { display: block; height: 100%; min-width: 2px; } +.prompt-staircase-seg { display: block; height: 100%; min-width: 2px; flex-shrink: 0; } .prompt-staircase-seg.is-cached { background-color: var(--violet-3); } .prompt-staircase-seg.is-replayed { background-color: var(--violet-2); box-shadow: inset 0 0 0 1px rgba(var(--violet-rgb), 0.25); } .prompt-staircase-seg.is-new { background-color: var(--violet); } diff --git a/styles/transcript.css b/styles/transcript.css index 8a0c537..73d2c30 100644 --- a/styles/transcript.css +++ b/styles/transcript.css @@ -325,6 +325,24 @@ font-variant-numeric: tabular-nums; } +.user-message .token-usage { + line-height: 1.35; + white-space: normal; +} + +.reply-token-count { + display: inline; + margin-left: 0.6em; + font-family: var(--font-geist-mono), ui-monospace, SFMono-Regular, Menlo, monospace; + font-size: var(--fs-1); + font-variant-numeric: tabular-nums; + line-height: 1.75; + color: var(--muted); + user-select: none; + pointer-events: none; + white-space: nowrap; +} + .message-aborted-note { margin-top: 4px; font-size: var(--fs-1);