@yanib/context-budget

Fit chat history into any model's context window.

Every chat app eventually writes the same code: estimate tokens, keep recent turns verbatim, compress the old ones, reserve room for the response, and handle the overflow error anyway. This package is that code — extracted from a production on-device assistant where the context windows are small and an overflow doesn't return a 400, it crashes the runtime.

Zero dependencies. No AI inside — the summarizer is a callback you can point at your model, with a deterministic extractive fallback built in. Works in Node, browsers, React Native, and edge runtimes; ships ESM + CJS. Also available for Python as llm-context-budget (same context_budget module name).

import { packMessages } from "@yanib/context-budget";

const { messages, summary, summarizedCount, fits } = await packMessages(history, {
  contextTokens: 8192,          // your model's window
  responseReserve: 600,         // headroom for the reply
  summary: conversation.summary,             // carried from the last turn
  alreadySummarized: conversation.cursor,    // …so old turns aren't re-folded
  systemBlocks: [personaPrompt, ragContext], // these consume budget too
  summarize: (prev, dropped) =>              // optional: bring your model
    llm.generateText(foldPrompt(prev, dropped)),
});

const reply = await llm.chat(messages);
conversation.summary = summary;      // persist for next turn
conversation.cursor = summarizedCount;

How it packs

1. The most recent turns stay verbatim (window size adapts to the model: 8 turns at 4k context, 16 at 8k, 24 at 16k+ — or set your own).
2. Older turns are represented by a running summary, carried between turns via summary + summarizedCount so nothing is summarized twice.
3. If the verbatim window still blows the budget, it's trimmed from the front and the trimmed turns are folded into the summary — one summarizer call per pack, not one per message.
4. Final guard: if a single message + system blocks still overflow, the summary is hard-truncated into whatever room remains. If even that can't fit, you get fits: false and you decide (the messages are still returned).

System blocks and the summary are merged into one system message — providers like Apple's Foundation Models accept only a single instructions block, and every other provider tolerates it.

Token estimation

The default estimator is the ~4-chars-per-token heuristic — deliberately dependency-free (real tokenizers cost megabytes and vary per model, and budgets carry headroom anyway). Have exact counts? Plug them in:

import { encode } from "gpt-tokenizer";
await packMessages(history, { estimateTokens: (t) => encode(t).length });

Also exported: createCharEstimator(ratio), truncateToTokens(text, max, est?, "head" | "tail"), estimateTokensOf, contextLimitsFor.

The overflow retry

Estimates are estimates. When the provider still throws, detect it and retry aggressively — half the verbatim window:

import { isContextOverflowError } from "@yanib/context-budget";

try {
  return await llm.chat(messages);
} catch (err) {
  if (!isContextOverflowError(err)) throw err;
  const retry = await packMessages(history, { ...options, aggressive: true });
  return await llm.chat(retry.messages);
}

The summarizer seam

summarize(previousSummary, droppedTurns) may be sync or async, and may call anything:

• Default: createExtractiveSummarizer() — one compact labeled line per dropped turn, capped total size keeping the tail. Deterministic, instant, offline.
• Your model: semantic summaries when quality matters. If your call fails, return previousSummary — a memory hiccup should never block a chat turn.

Extra message fields (images, ids, timestamps) pass through packing untouched, so multimodal and app-specific metadata survive.

API

packMessages(history, options?) → Promise<{
  messages,          // one merged system message (if any) + verbatim window
  summary,           // persist and pass back next turn
  summarizedCount,   // pass back as alreadySummarized next turn
  dropped,           // turns folded into the summary this call
  usedTokens, budget, fits,
}>

All options: contextTokens (4096), responseReserve (600), recentWindow (auto), aggressive, estimateTokens, summary, alreadySummarized, systemBlocks, summarize.

License

MIT © Binaya Dhakal