Skip to content

fix(proxy): a lost stdout/stderr reader must not wedge the proxy - #333

Closed
codeslake wants to merge 1 commit into
cnighswonger:mainfrom
codeslake:fix/stdio-epipe-wedges-proxy
Closed

fix(proxy): a lost stdout/stderr reader must not wedge the proxy#333
codeslake wants to merge 1 commit into
cnighswonger:mainfrom
codeslake:fix/stdio-epipe-wedges-proxy

Conversation

@codeslake

Copy link
Copy Markdown
Contributor

What broke

A live proxy stopped carrying traffic for 27 minutes. It was not down — it was pegged at 100% CPU, accepting connections and answering none, while /health kept returning 200 in 0.35s with every self-reported field green (https_proxy=127.0.0.1:8118, https_proxy_measured=true).

The trigger was ordinary: a leftover ... | tee <file> wrapper was killed. That tee was the only reader of the pipe the proxy holds as stdout/stderr.

pipe holders after the tee died:  holder fd1, holder fd2, proxy fd2
                                  -> writers only, zero readers

Why one dead log reader takes the proxy down

sample on the wedged process, verbatim shape:

TriggerUncaughtException
  -> MessageHandler::ReportMessage
    -> ErrorStackGetter -> GetFormattedStack -> FormatStackTrace
      -> node::PrepareStackTraceCallback -> v8::Function::Call -> ...

The cycle:

  1. a write to the reader-less pipe raises EPIPE
  2. EPIPE on a stream arrives as an asynchronous 'error' event, not a synchronous throw, and with no listener Node promotes it to uncaughtException
  3. installSelfHeal's handler answers uncaughtException by formatting err.stack and writing it to that same dead stderr
  4. → 1

The mechanism that exists to keep the proxy up is what takes it down. A try/catch around the write cannot help, because step 2 is not a throw.

Measured: 22 minutes of CPU time burned, 18 connections accepted, none answered.

The fix

  • 'error' listeners on process.stdout / process.stderr that swallow EPIPE and ERR_STREAM_DESTROYED, so the async half never reaches uncaughtException.
  • The self-heal handlers' own logging goes through a guarded write, so a log line can never become the next uncaught exception. Both halves are needed: the guard covers a synchronous throw on a destroyed stream, the listeners cover the asynchronous event.
  • Both listeners are removed in removeSelfHeal, matching what that function already does for the other two — an embedded host process that ran forward mode earlier must get Node's default behaviour back, not keep an EPIPE swallower installed by a proxy that has closed.

Losing a log reader is ordinary — a closed terminal, a rotated file, a killed tee. It should cost the log line and nothing else.

Tests

test/proxy-stdio-epipe.test.mjs reproduces the outage rather than simulating it: it removes the only reader of a child proxy's stdio, raises an uncaught exception from a check-phase callback (where the real one came from), then asserts the proxy still serves and does not spin.

  • Full suite on this commit: 1783 tests, 1782 pass, 0 fail, 1 skipped.
  • Mutation-checked: deleting the two listeners fails the test with burned 2.00s of CPU in 1.5s wall; restoring them passes. Both directions run and recorded.
  • Note the /health assertion inside the test passes even in the broken state — independently reproducing the field observation that /health is not a liveness signal for this failure.

Non-Functional Requirements

  • Size/complexity budget — ~20 lines of production code in one existing function, plus one test and one fixture. Landed at 38 added lines in proxy/server.mjs.
  • Threat model — unchanged. No new inputs, no new network surface, no credential handling. The listeners only discard error objects for two error codes and log the rest.
  • Maintainability constraints — no new abstraction, no new module, no config knob. safeWrite is a 1-line local closure with 3 call sites in the same function; inlining it three times would be worse.
  • Performance/reliability — this is the reliability fix. Two listeners on process stdio, no per-request cost.
  • Load-bearing?yes. It changes process-wide uncaught-exception behaviour for the forward-proxy path.

Note for reviewers

Branched from upstream/main and self-contained on it. An earlier draft routed the handlers through a say() helper — that helper does not exist on main (it lives on another of our branches), and the resulting ReferenceError: say is not defined inside the uncaughtException handler killed the proxy outright. Caught by the test before commit; the landed version depends on nothing outside this diff.

🤖 Generated with Claude Code

When the last reader of the proxy's stdout/stderr pipe goes away — a closed
terminal, a rotated log, a killed `tee` — the next write raises EPIPE. That
arrives as an asynchronous 'error' event, not a synchronous throw, so Node
promotes it to uncaughtException. The self-heal handler then formats err.stack
and writes it to the same dead stderr, raising the next EPIPE: the mechanism
that exists to keep the proxy up is what takes it down.

Measured on a live proxy: 100% CPU, 22 minutes of CPU time burned, 18
connections accepted and none answered — while /health kept returning 200 in
0.35s with every self-reported field green, so no health-based check saw it.

Install 'error' listeners on process.stdout/stderr that swallow EPIPE, and
route the self-heal handlers' own logging through a guarded write so a log
line can never become the next uncaught exception. Both halves are needed: the
try/catch covers a synchronous throw on a destroyed stream, the listeners cover
the asynchronous event that caused the outage. Removed again in removeSelfHeal,
so an embedded host process regains Node's default behaviour.

The test reproduces the outage rather than simulating it: it removes the only
reader of the child's stdio, raises an uncaught exception from a check-phase
callback, then asserts the proxy still serves and does not spin.
Mutation-checked — deleting the two listeners fails it with
"burned 2.00s of CPU in 1.5s wall".

Co-Authored-By: Claude <noreply@anthropic.com>
@codeslake

Copy link
Copy Markdown
Contributor Author

Closing — this belongs inside #304, not as a separate PR. Moving the commit onto that branch.

— Proxy Builder

@codeslake codeslake closed this Aug 14, 2026
@codeslake
codeslake deleted the fix/stdio-epipe-wedges-proxy branch August 14, 2026 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant