Health check Expected Status Code collides between two resources sharing the same backend host:port on reconnect

[gravis]

Describe the Bug

When two resources share the same backend host:port but have different health-check Expected Status Codes, the server sends conflicting hcStatus values for the same target across two messages on a reconnect, and the resource ends up permanently Unhealthy even though it responds correctly.

Newt applies each health-check config it receives faithfully, keyed by target ID — so this is a server-side payload inconsistency, not a Newt bug. (Originally filed against Newt as fosrl/newt#411; traced to the server.)

Setup that triggers it

• Resource A — health check GET /manifest.json, Expected Status 200 → backend 10.0.0.10:8123
• Resource B — health check GET /api/mcp, Expected Status 401 → same backend 10.0.0.10:8123

Clean boot is fine. After a Newt websocket reconnect (server restart, DNS blip, or brief network drop), Resource A goes Unhealthy and stays that way until Newt is restarted.

Root cause (from Newt logs on reconnect)

On reconnect the server sends two messages that disagree on target A's expected code:

Websocket connected
Received sync message
Syncing health check targets: 2 desired targets
Sync complete: removed 0, added 0, updated 2 targets
Target B initial status: healthy
Target A initial status: healthy          <-- sync payload: target A expects 200 (correct)
...
Replacing existing target with ID A       <-- newt/wg/connect (registration) payload
Replacing existing target with ID B
Target B initial status: healthy
Target A initial status: unhealthy        <-- registration payload: target A expects 401 (wrong)

Immediately after, Newt logs every interval:

Target A: unexpected status code: 200 (expected: 401)

So:

• the newt/sync payload carries target A → hcStatus: 200 (correct)
• the newt/wg/connect registration payload carries target A → hcStatus: 401 (wrong — it's Resource B's expected code)

The registration message arrives second and overwrites the correct value. Target A (/manifest.json) then fails forever because it's checked against 401.

This strongly suggests the server builds the health-check target list (at least in the registration/getConfig path) keyed or de-duplicated by backend host:port, so when two resources share a backend, one resource inherits the other's expected status code.

Expected Behavior

Each health-check target should carry its own configured Expected Status Code regardless of whether another resource shares the same backend host:port, and the sync and registration payloads should agree.

Environment

• Pangolin Version: 1.20.0 (also observed on 1.19.4)
• Newt Version: 1.14.0

Workaround

Restart Newt for a clean reload — temporary; the collision returns on the next reconnect. Or avoid two resources sharing an identical backend host:port for health checks (disable the health check on one, or differentiate the backends).

[gravis]

Closing — after tracing this in the newt source, the root cause is client-side, not server-side. With a single database, both the newt/sync and newt/wg/connect payloads are built by the same deterministic buildTargetConfigurationForNewtClient, so the server cannot emit different hcStatus for a target between them.

The actual bug: newt's handleConnect decodes the registration payload into a reused n.wgData, and Go's json array-into-existing-slice merge (with null being a no-op for non-pointer ints) lets a target inherit a stale hcStatus from a previous connection when the health-check rows reorder on reconnect. Fixed in the client: fosrl/newt#413.

One minor server-side hardening that would make this class of bug less likely: add a stable ORDER BY to the health-check target query in buildTargetConfigurationForNewtClient so the array order is deterministic across reconnects. Not required for the fix (the client change resolves it), but happy to open a small PR if you'd want it.