Skip to content

Commit 592fdc8

Browse files
committed
fix(selfhost): attach a pg.Pool error listener to prevent uncaught-exception crashes
GITTENSORY-1R/1S ("uncaughtException: terminating connection due to administrator command") and GITTENSORY-1T ("boot: Postgres not ready after 30000ms") are one incident, traced on the same container within a 29-minute window: Postgres itself was restarted while the app held live pooled connections. node-postgres crashes the WHOLE process with an uncaught exception if a Pool has no "error" listener and an idle client's connection drops -- Node's EventEmitter throws on an unhandled "error" event. The crash then triggered a boot-retry crash loop (waitForPostgres's 30s wait, repeatedly failing since Postgres was mid-restart) until Postgres was fully back up. Attach pool.on("error", ...) right after the pool is constructed. The pool already removes a broken client and opens a fresh one on the next checkout on its own -- the only thing missing was a listener so Node stops treating an idle client's connection-level error as unhandled. Logs via the same console.error JSON convention used throughout this codebase so the failure still reaches Sentry/Loki, just without taking the process down.
1 parent 46581d7 commit 592fdc8

1 file changed

Lines changed: 16 additions & 0 deletions

File tree

src/server.ts

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,22 @@ async function buildPostgresBackend(
218218
const pg = (await import("pg")).default;
219219
pg.types.setTypeParser(20, (v: string) => Number.parseInt(v, 10)); // int8 (COUNT) → number, like D1
220220
const pool = new pg.Pool({ connectionString: url, max: resolvePostgresPoolMax() });
221+
// node-postgres crashes the WHOLE process with an uncaught exception if the pool has no "error" listener and
222+
// an IDLE client's connection drops (Node's EventEmitter throws on an unhandled "error" event) -- confirmed
223+
// live (GITTENSORY-1R/1S): Postgres itself being restarted ("terminating connection due to administrator
224+
// command") took the whole app down, which then crash-looped for ~29 minutes hitting waitForPostgres's 30s
225+
// boot timeout (GITTENSORY-1T) until Postgres was fully back up. The pool already removes a broken client and
226+
// opens a fresh one on the next checkout on its own -- the only thing missing was a listener so Node stops
227+
// treating an idle client's connection-level error as unhandled.
228+
pool.on("error", (error) => {
229+
console.error(
230+
JSON.stringify({
231+
level: "error",
232+
event: "selfhost_pg_pool_error",
233+
message: error instanceof Error ? error.message : "unknown error",
234+
}),
235+
);
236+
});
221237
const db = createPgAdapter(pool);
222238
const queue = createPgQueue(pool, consume);
223239
await queue.init();

0 commit comments

Comments
 (0)