Skip to content

Avoid passing non garbage collectable composite signal to tool call - #27986

Merged
ykmsd merged 3 commits into
mainfrom
ac-listener-memory-leak
Jun 26, 2026
Merged

Avoid passing non garbage collectable composite signal to tool call#27986
ykmsd merged 3 commits into
mainfrom
ac-listener-memory-leak

Conversation

@ykmsd

@ykmsd ykmsd commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Description

This PR is my attempt to fix memory leak in agent loop worker.

I have multiple snapshots from agent loop worker, and most of them are retained by gcPersistentSignals. For example in one of my snapshots:

- Total heap self-size: ~1856 MB
- Retained by gcPersistentSignals: ~753 MB (40.6%)

Below is Yuka's current understanding of what is happening, I might be wrong so be extra careful when you read it 🫡

What is an AbortController/AbortSignal?

AbortController is the way to cancel fetch requests or other asynchronous operations.

For example, the following fetch will be cancelled if it doesn't finish within 5000ms:

const controller = new AbortController();
const signal = controller.signal;

setTimeout(() => controller.abort(), 5000);

fetch(url, { signal }).then(response => {
    return response.text();
}).then(text => {
    console.log(text);
});

(note: you can now also use AbortSignal.timeout instead of setTimeout: await fetch(url, { signal: AbortSignal.timeout(5000) }))

You can attach an event listener to the signal:

signal.addEventListener('abort', () => {
    console.log(signal.aborted); // logs true
});

Why there is a memory leak and what is gcPersistentSignals?

There is a bug in MCP sdk, they attach an event listener to the abort signal but never remove it, and also it doesn't have { once: true } (meaning that even if it's aborted the listener will not be removed, you can read more about once here).

https://github.com/modelcontextprotocol/typescript-sdk/blob/v1.27.1/src/shared/protocol.ts#L1202-L1204

            options?.signal?.addEventListener('abort', () => {
                cancel(options?.signal?.reason);
            });

The event listener they attach captures cancel, which carries a reject function and it is a closure that points back at the whole Promise. A settled promise retains its fulfillment value ([[PromiseResult]]) in V8, because the abort listener is never removed, reject stays reachable indefinitely, and the result of every past tool call accumulates in memory for the lifetime of the pod.

This bug is fixed in v2 alpha version but not in v1. If you use a single abort signal and it's garbage collectable, this bug is harmless, because once V8 cannot reach the abort signal, everything (the listener, handler, and its closure) will be garbage collected.

However, the problem is we pass a composite signal and one of them is non garbage collectable:

https://github.com/dust-tt/dust/blob/main/front/temporal/agent_loop/activities/run_tool.ts#L210-L214

  const abortSignal = AbortSignal.any([
    Context.current().cancellationSignal,
    getShutdownSignal(),
  ]);

getShutdownSignal is a singleton that is declared at module level which will never be garbaged collected during the entire pod lifecycle.

When you call AbortSignal.any([toolCallSignal, getShutdownSignal()]), Node.js wires up two pointers between each source and the composite signal:

        ┌──────────── WeakRef ───────────►┐
   source signal                      composite signal
        ◄─────────── WeakRef ────────────┘

It's a WeakRef, so even if one of the sources is still alive, the composite signal can be garbage collected once nothing strongly references it.

However, if you add an event listener to a composite signal, you don't want the signal to be garbage collected (otherwise nothing happens when abort is fired). So Node.js will add a composite signal to gcPersistentSignals while it has an abort listener AND still has at least one live source. This will strongly reference the signal:

https://github.com/nodejs/node/blob/ccdfb374383a3b0126693089780314f967881d20/lib/internal/abort_controller.js#L238-L252

  [kNewListener](size, type, listener, once, capture, passive, weak) {
    super[kNewListener](size, type, listener, once, capture, passive, weak);
    const isTimeoutOrNonEmptyCompositeSignal = this[kTimeout] || (this[kComposite] && this[kSourceSignals]?.size);
    if (isTimeoutOrNonEmptyCompositeSignal &&
        type === 'abort' &&
        !this.aborted &&
        !weak &&
        size === 1) {
      // If this is a timeout signal, or a non-empty composite signal, and we're adding a non-weak abort
      // listener, then we don't want it to be gc'd while the listener
      // is attached and the timer still hasn't fired. So, we retain a
      // strong ref that is held for as long as the listener is registered.
      gcPersistentSignals.add(this);
    }
  }

It will be removed from gcPersistentSignals only when the listener count drops back to 0 or both sources are gone. It will never happen in our case because sdk doesn't remove the listener when it's completed or aborted, and one of sources is alive forever.

So the retention chain is:
gcPersistentSignals → composite signal → abort listener → cancel closure → reject → Promise → [[PromiseResult]]

How can we fix it? The MCP SDK attaches an abort listener that we have no way to remove. So instead of handing it the composite signal, we hand it a signal from a locally scoped AbortController. Because that signal is a non-composite signal, Node.js never adds it to `gcPersistentSignals` and because it's only referenced locally, it is garbage collected once the tool call ends (along with the SDK's listener and the retained tool result). if the composite signal receives an abort event (pod shutdown or workflow cancellation), our bridge listener forwards the abort, with its reason, to the per-call AbortController, so the in-flight tool call is still cancellable (I think??).

Tests

Risk

Deploy Plan

@vercel

vercel Bot commented Jun 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated (UTC)
playground Ignored Ignored Preview Jun 26, 2026 2:36pm
storybook Ignored Ignored Preview Jun 26, 2026 2:36pm

Request Review

@ykmsd ykmsd changed the title abort controller imp Avoid passing non garbage collectable composite signal to tool call Jun 25, 2026
@ykmsd
ykmsd marked this pull request as ready for review June 25, 2026 21:02

@dust-agent dust-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coding Rules LGTM \o/

@ykmsd
ykmsd requested review from flvndvd and matteotrab June 25, 2026 21:17

@flvndvd flvndvd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯 / 💯


if (compositeSignal.aborted) {
perToolCallController.abort(compositeSignal.reason);
return callTool(perToolCallController.signal);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If plan is to let MCP handles the abort signal can we add a comment right above this line, please?

@ykmsd

ykmsd commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

I tested it locally (called wait tool and then sent a signal to mimic shutdown) and it seems to be working. I will merge it and monitor 🫡

Also note that most likely we need more work in memory management, I'm not sure how much of spikes we saw is related to this memory leak because we seem to load something huge like mp4, pptx etc. I will continue my investigation 🫡

@ykmsd
ykmsd merged commit da718ba into main Jun 26, 2026
40 checks passed
@ykmsd
ykmsd deleted the ac-listener-memory-leak branch June 26, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants