conversation-outline rebuilds the full history with every tool output on each appended event; 1.3 s event-loop stalls on a 24 MB thread

[SawyerHood]

Summary

While one thread streamed, the bb server logged 11 Event loop stalled warnings in 24 minutes, with max stalls of 1.2–1.3 s. The slowest work was GET /api/v1/threads/:id/conversation-outline for the thread with the most event data (24 MB at the time). The outline route rebuilds the full-history outline from scratch on every appended event because its cache key is ${threadId}:${maxSeq}, and the rebuild reads every toolCall item/completed row with its full inline output, synchronously on the event loop. I expected the outline to cost a few milliseconds per event batch, not hundreds of milliseconds of blocking SQLite time that freezes every other client.

Versions and environment

• bb 0.38.0 (bb --version), packaged app served from source checkout ~/projects/bb at a108fa7ef383a99cbc3c56388bb4fdf345db1492 (server started 2026-08-18 14:49, PID 3590098, Node v24.18.0)
• Code cited below at main = fc6e11fe771d61dd854127e3333a0f4d46b7519f (same code paths; unchanged since Full-history conversation outline for the thread TOC minimap #399/Improve open in editor remote support #403)
• Ubuntu 26.04 LTS, Linux 7.0.0-29-generic, 16 cores, 58 GB RAM
• ~/.bb/bb.db is 3.0 GB, 980,873 event rows, 2,656 threads, WAL 58 MB
• Viewed remotely through bb Connect (bee.getbb.app); streaming thread used the claude-code provider in a worktree environment

Steps to reproduce

Observed in production (the 24 MB thread has since been truncated to 2.5 MB by something outside this report, so the 1.3 s figure is no longer live; the mechanism still reproduces):

1. Pick the thread with the most outline-relevant bytes:

   sqlite3 ~/.bb/bb.db "select thread_id, count(*), sum(length(data)) b from events where (type in ('client/turn/requested','turn/input/accepted','turn/started','turn/completed','system/manager/user_message','system/thread/interrupted','system/error','provider/error','item/agentMessage/delta','item/plan/delta')) or (type='item/completed' and item_kind in ('agentMessage','plan')) or (type in ('item/started','item/completed','item/backgroundTask/progress','item/backgroundTask/completed') and item_kind in ('backgroundTask','toolCall')) group by thread_id order by b desc limit 3;"

   On my DB today: thr_62bp724ett|3662|5670143 (5.7 MB).
2. Time a cold then warm outline request (the cache key is threadId:maxSeq, so the first call after any new event is cold):

   for i in 1 2 3; do curl -s -o /dev/null -w "ttfb=%{time_starttransfer}s bytes=%{size_download}\n" http://127.0.0.1:38886/api/v1/threads/thr_62bp724ett/conversation-outline; done

3. Open that thread in the wide layout (so the TOC is visible) and start a turn that runs many tool calls with large outputs (for example cat a few big files). Watch the log:

   tail -f ~/.bb/logs/server.$(ls ~/.bb/logs | command grep -o 'server\.[0-9]*' | sort -t. -k2 -n | tail -1 | cut -d. -f2).log | command grep -a 'Event loop stalled\|conversation-outline'

   Every events-appended batch invalidates the outline query on the client, the client refetches, the server misses the cache, and a full rebuild runs.

To reproduce at the original scale without waiting, pnpm seed:perf -- --reset --events 400000 and pick the largest thread the same way; I did not run this (unverified).

Did not reproduce with: a small thread (thr_qvau3b2d5b, 1.1 KB outline → 0.6–33 ms), or a warm cache (second call 1–3 ms). The timeline route for the same 24 MB thread stayed at median 69 ms / max 368 ms in the same window, so this is specific to the outline path.

Expected vs actual

Actual — ~/.bb/logs/server.66.log, 2026-08-19 22:50–23:14 UTC-7 (trimmed; slowestWork names the request that held the loop):

{"time":1787180467591,"maxDelayMs":748.2,"p99DelayMs":705.2,"lastWork":"GET /api/v1/threads/thr_n3xqyz69nk/timeline","lastWorkMs":700.1,"slowestWork":"GET /api/v1/system/execution-options","slowestWorkMs":2701.2,"msg":"Event loop stalled"}
{"time":1787180478187,"maxDelayMs":1159.7,"p99DelayMs":430.7,"slowestWork":"GET /api/v1/threads/thr_n3xqyz69nk/conversation-outline","slowestWorkMs":1155.2,"msg":"Event loop stalled"}
{"time":1787180488220,"maxDelayMs":1205.9,"p99DelayMs":1205.9,"slowestWork":"GET /api/v1/threads/thr_n3xqyz69nk/conversation-outline","slowestWorkMs":1192.3,"msg":"Event loop stalled"}
{"time":1787180493293,"maxDelayMs":1329.6,"p99DelayMs":1329.6,"slowestWork":"GET /api/v1/threads/thr_n3xqyz69nk/conversation-outline","slowestWorkMs":1309.6,"msg":"Event loop stalled"}
{"time":1787180689043,"maxDelayMs":559.9,"slowestWork":"GET /api/v1/threads/thr_n3xqyz69nk/conversation-outline","slowestWorkMs":552.3,"msg":"Event loop stalled"}

The same window logged the outline's UNION ALL query as slow 18 times (100–266 ms each, 2,654 ms total):

{"time":1787180491958,"bindingArgumentCount":22,"durationMs":240.6,"operation":"all","sql":"select \"created_at\", \"data\", \"id\", \"item_id\", \"item_kind\", \"provider_thread_id\", \"scope_kind\", \"sequence\", \"thread_id\", \"turn_id\", \"type\" from \"events\" where (\"events\".\"thread_id\" = ? and \"events\".\"type\" in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) union all ..."}

Connect's per-request log for the same thread: 166 outline requests in 24 minutes (up to 29/min), avg TTFB 212 ms, max 1,512 ms, response 3 KB gzip.

Today (thread now 5.7 MB of outline-relevant rows, step 2 above):

thr_62bp724ett: ttfb=0.163744s bytes=39640   <- cold
thr_62bp724ett: ttfb=0.043309s bytes=39640
thr_62bp724ett: ttfb=0.003141s bytes=39640   <- warm

Expected: an outline refresh during streaming should not block the server for more than a few milliseconds, and the server should not re-read megabytes of tool output to produce a 3 KB table of contents.

Evidence

• Thread ids: stalled thread thr_n3xqyz69nk (project proj_atixi2qwed); live measurement thread thr_62bp724ett; investigation thread thr_272jbuyzju.
• Stall counts per 10 MB log file: server.61 0, .62 3, .63 6, .64 0, .65 0, .66 11 (24 min) — the spike coincides with the streaming turn on thr_n3xqyz69nk.
• Why the rebuild is expensive: the outline query selects full data for every toolCall item/started and item/completed row. On thr_62bp724ett, toolCall item/completed rows are 3.42 MB of the 5.67 MB read (60%); the outline only needs their lifecycle/grouping, not the tool output.

  toolCall|item/completed|667|3424767
  toolCall|item/started|667|561353
  agentMessage|item/completed|736|540959
  backgroundTask|item/started|533|303384
  

• Code (at fc6e11fe7):
  • Outline query, three full-row selects unioned:

    bb/packages/db/src/data/events.ts

    Lines 2519 to 2590 in fc6e11f

    	export function listStoredConversationOutlineEventRows(
    	db: DbConnection,
    	args: ListStoredConversationOutlineEventRowsArgs,
    	): StoredEventRow[] {
    	const lifecycleTypes = [
    	"client/turn/requested",
    	"turn/input/accepted",
    	"turn/started",
    	"turn/completed",
    	"system/manager/user_message",
    	"system/thread/interrupted",
    	"system/error",
    	"provider/error",
    	"item/agentMessage/delta",
    	"item/plan/delta",
    	] satisfies ThreadEventType[];
    	const conversationItemKinds = [
    	"agentMessage",
    	"plan",
    	] satisfies ThreadEventItemType[];
    	const structuralItemKinds = [
    	"backgroundTask",
    	"toolCall",
    	] satisfies ThreadEventItemType[];
    	const structuralItemLifecycleTypes = [
    	"item/started",
    	"item/completed",
    	"item/backgroundTask/progress",
    	"item/backgroundTask/completed",
    	] satisfies ThreadEventType[];
    	const lifecycleRows = db
    	.select(storedEventRowFields)
    	.from(events)
    	.where(
    	and(
    	eq(events.threadId, args.threadId),
    	inArray(events.type, lifecycleTypes),
    	),
    	);
    	const completedConversationRows = db
    	.select(storedEventRowFields)
    	.from(events)
    	.where(
    	and(
    	eq(events.threadId, args.threadId),
    	eq(events.type, "item/completed"),
    	inArray(events.itemKind, conversationItemKinds),
    	),
    	);
    	const structuralRows = db
    	.select(storedEventRowFields)
    	.from(events)
    	.where(
    	and(
    	eq(events.threadId, args.threadId),
    	inArray(events.type, structuralItemLifecycleTypes),
    	inArray(events.itemKind, structuralItemKinds),
    	isNotSupersededBackgroundTaskProgress,
    	),
    	);
    	// These disjoint branches let SQLite use the thread/type/item-kind indexes.
    	// A single OR plus SQL ordering makes it scan every event through the
    	// thread/sequence index instead, which dominates cold loads of long threads.
    	const rows = unionAll(
    	lifecycleRows,
    	completedConversationRows,
    	structuralRows,
    	).all();
    	return rows.sort((left, right) => left.sequence - right.sequence);
    	}

  • The timeline window query already has storedEventRowFieldsWithInlineOutputLimit to cap inline output inside SQLite; the outline query does not use it:

    bb/packages/db/src/data/events.ts

    Line 966 in fc6e11f

    function storedEventRowFieldsWithInlineOutputLimit(

  • Route cache keyed on threadId:maxSeq, so every appended event is a miss:

    bb/apps/server/src/routes/threads/data.ts

    Lines 422 to 433 in fc6e11f

    	get(routes.conversationOutline, (context) => {
    	const thread = requirePublicThread(deps.db, context.req.param("id"));
    	const maxSeq = getLatestThreadSequence(deps.db, { threadId: thread.id });
    	const cacheKey = `${thread.id}:${maxSeq}`;
    	const cached = conversationOutlineCache.get(cacheKey);
    	if (cached !== undefined) {
    	// Re-insert to mark most-recently-used.
    	conversationOutlineCache.delete(cacheKey);
    	conversationOutlineCache.set(cacheKey, cached);
    	return context.json(cached);
    	}
    	const response = buildThreadConversationOutline(deps.db, thread, {

  • Rebuild runs inside runEventLoopWorkSync:

    bb/apps/server/src/services/threads/timeline.ts

    Lines 1899 to 1908 in fc6e11f

    	export function buildThreadConversationOutline(
    	db: DbConnection,
    	thread: Thread,
    	options: BuildThreadConversationOutlineOptions,
    	): ThreadConversationOutlineResponse {
    	return runEventLoopWorkSync(`conversation-outline ${thread.id}`, () => {
    	const rawEventRows = listStoredConversationOutlineEventRows(db, {
    	threadId: thread.id,
    	});
    	const decodedRawEvents = rawEventRows.map((row) =>

  • Client invalidates the outline on every realtime events-appended batch:

    bb/apps/app/src/hooks/cache-owners/cache-invalidation-groups.ts

    Lines 131 to 145 in fc6e11f

    	threadId,
    	}: ThreadScopedInvalidationArgs): QueryKey[] {
    	return threadId
    	? [
    	threadTimelineQueryKeyPrefix(threadId),
    	threadConversationOutlineQueryKeyPrefix(threadId),
    	]
    	: [
    	allThreadTimelineQueryKeyPrefix(),
    	allThreadConversationOutlineQueryKeyPrefix(),
    	];
    	}
    	export function getThreadQueueContentInvalidationQueryKeys({
    	threadId,

    via

    bb/apps/app/src/hooks/cache-owners/realtime-cache-registry.ts

    Lines 898 to 905 in fc6e11f

    	function dirtyThreadTimelineQueries({
    	eventTypes,
    	queryClient,
    	threadId,
    	}: ThreadRealtimeDirtyContext): void {
    	// Window only: completed turn-summary-details are immutable, so realtime
    	// event batches must not refetch open detail panels (see helper docs).
    	const queryKeys = getThreadTimelineWindowInvalidationQueryKeys({ threadId });

• Measurement method: stall and slow-query lines come from the server's own attribution logger (Attribute event-loop stalls to the work that blocked them #1437); request timings from the [plugin:connect] bb connect thread load lines and curl -w; byte counts from sqlite3 against the live DB.

What you ruled out

• Not Synchronous SQLite on the event loop: one cold query froze all clients for 11s on a 1.7GB bb.db #1131 / Thread-list reads still run synchronously on the event loop after #1204 #1207: those cover thread-list and cold-query stalls; neither mentions the outline route, and the outline stalls here are hot-path and repeat per event batch.
• Not Thread timeline never unmounts history: 22k DOM nodes and 250-600ms long tasks after scrolling a large thread #1301 (frontend DOM growth) or Opening a thread fires ~19 API requests, including per-open environment git probes #1303 (request fan-out on thread open): this is a server-side synchronous rebuild on every appended event after the thread is already open.
• Not the timeline route: same thread, same window, median 69 ms.
• Not DB pragmas: Set SQLite cache, mmap, and WAL synchronous pragmas #1438 is already in the running build; the work is proportional to bytes read, not cache misses.
• Still present on main at fc6e11fe7 (code unchanged since Full-history conversation outline for the thread TOC minimap #399/Improve open in editor remote support #403).

Suggested fix

Any one of these removes most of the cost; the first two together are the smallest change:

1. In listStoredConversationOutlineEventRows, select storedEventRowFieldsWithInlineOutputLimit(<small cap>) instead of storedEventRowFields so SQLite truncates $.item.aggregatedOutput / $.item.result / $.item.resultText before the rows cross into JS. The outline never renders tool output.
2. Make the outline incremental: cache by threadId with the maxSeq it was built at, and on a request with a higher maxSeq read only rows with sequence > cachedMaxSeq and fold them into the cached outline (the timeline route already does a delta for afterSequence).
3. On the client, debounce outline refetches during streaming (for example once per N seconds while events-appended batches arrive) instead of on every batch; the TOC does not need sub-second freshness.
4. If the rebuild stays synchronous, add the outline query to packages/db/test/query-plans.test.ts with a bytes-read assertion so a future field addition does not regrow it.

Suggested priority and effort

Medium-High — hits any user who keeps a long, tool-heavy thread open in the wide layout while it streams; every other client on that server stalls for up to ~1.3 s per event batch; no workaround short of closing the thread. Effort Low for (1)+(3), Medium for (2).

Investigation thread: thr_272jbuyzju.

AGENT GENERATED: by Claude Opus 5