Newt site WG handshake never completes after upgrading server to 1.20.0; gerbil 1.4.2 also appears to regress registration freshness

[willyp713]

Describe the Bug

Symptom:

Pull of pangolin:latest (1.19.x → 1.20.0, and gerbil 1.4.1 → 1.4.2) and newt site's WireGuard data plane never comes up. The control websocket connects, but the WG handshake never completes, so every proxied resource returns 504 from Traefik. The site loops connect → ping-timeout → reconnect every ~3 minutes. Full stack stop/start and newt restarts (fresh WG keys each time) and a full reboot of the server host (ruling out stale conntrack / docker iptables state) do not help.

Observed evidence:

Pangolin refuses site registration on staleness grounds (with gerbil 1.4.2):

pangolin | [warn]: Site last hole punch is too old; skipping this
register. The site is failing to hole punch and identify its network
address with the server.

Newt (1.14.0, default netstack mode) cycles:

INFO:  Websocket connected
INFO:  Server version: 1.20.0
INFO:  Started tcp proxy to <LAN_TARGET>:<PORT>   (multiple targets, all healthy)
WARN:  Ping attempt N failed: failed to read ICMP packet: i/o timeout
WARN:  Periodic ping failed (4 consecutive failures): ...
WARN:  Connection to server lost after 4 failures. Continuous reconnection attempts will be made.
INFO:  Connecting to endpoint: pangolin.example.com
WARN:  Failed to start hole punch: hole punch already running

Gerbil re-adds the peer every cycle without a handshake landing:

gerbil | INFO: Clearing connections for added peer with WG IP: <WG_IP>
gerbil | INFO: Peer <redacted-pubkey>= added successfully   (every ~2m50s, forever)

After pinning gerbil back to 1.3.1, the "hole punch is too old" warnings stop appearing across subsequent reconnects, but the WG handshake still never completes on 1.20.0.

Suspected causes (two stacked problems):

1. gerbil 1.4.2's hole-punch dedup cache TTL (2.5 s) appears to race newt's ~2 s hole-punch interval (maybe related to Hole-punch endpoint dedup skips re-registration on reconnect → clients stuck "connected but not registered" (regression in 1.4.x) gerbil#96). If newt punches every ~2 s from an unchanged NAT endpoint, the cache would never expire between punches, notifyServer() would stay suppressed, and the server's "last hole punch" timestamp would go stale, at which point the freshness gate observed in 1.20.0 refuses registration indefinitely. Consistent with the warnings disappearing on gerbil 1.3.1 (because of no dedup cache?).

2. Even with the staleness warnings gone (by reverting to gerbil 1.3.1), the WG handshake still never completes on Pangolin 1.20.0 with the latest newt (1.14.0). So there appears to be a second, gerbil-version-independent problem in 1.20.0's site registration or WG peer setup.

Additional Context:

• Timeline: gerbil's logs show the peer-re-add churn beginning the same minute the upgraded containers started, with no healthy gaps since.
• Ruled out: endpoint DNS (the control WS resolves and connects every cycle), stale container/network state (full stack down/up, newt restarts with fresh keys, and a full OS reboot of the server host to rebuild conntrack and docker's iptables) and the site's local targets (all healthcheck healthy over the control channel).

Environment

• OS Type & Version: Debian 13 (server); site host runs newt in Docker on Linux
• Pangolin Version: 1.20.0 (upgraded from 1.19.x; sqlite migrated on first boot)
• Edition (Community or Enterprise): Community
• Gerbil Version: 1.4.2 at first, then pinned 1.3.1 (staleness warnings stopped; handshake fails on both)
• Traefik Version: 3.7.7
• Newt Version: 1.14.0 (docker fosrl/newt, default netstack mode, env config only)
• Client Version: N/A (no olm clients involved; affected path is the newt site tunnel)

Server layout is the standard docker compose install (gerbil publishes 51820/udp, 21820/udp, 443, 80; traefik in gerbil's network namespace). Site is behind consumer NAT.

To Reproduce

1. Start from a working Pangolin 1.19.x server (docker compose, :latest tags) with a newt 1.14.0 site in Docker behind NAT.
2. Redeploy the server stack so it pulls Pangolin 1.20.0 and gerbil 1.4.2.
3. Observe: newt's control websocket connects, but tunnel pings time out and it reconnects every ~3 minutes; Pangolin logs "Site last hole punch is too old; skipping this register" on registration attempts; all proxied resources 504.
4. Pin gerbil to 1.3.1 and redeploy; restart newt.
5. Observe: the "too old" warnings stop, but the WG handshake still never completes and resources still 504.

Expected Behavior

Upgrading the server to 1.20.0 should not strand a previously working newt 1.14.0 site: hole-punch freshness should survive gerbil 1.4.2's dedup cache, and the WG handshake should complete once registration succeeds.

[oschwartz10612]

You could have a private key mismatch error with the database if you have cleared out your key file from the config directory recently. Try to clear the exit nodes, do a full docker down, then up and let me know if things improve. If not, please place the full log dump of pangolin, gerbil, and newt for us to take a look. https://docs.pangolin.net/self-host/advanced/container-cli-tool#clear-exit-nodes

[willyp713]

Thanks for the clear-exit-nodes pointer earlier, it led me down a long troubleshooting path that ruled out the key/DB path along the way.

Root-cause seems to be rootless Docker not Pangolin/gerbil/newt changes.

What was wrong:

Rootless-Docker port-forwarding wasn't preserving the client source address for gerbil. Gerbil saw every packet as coming from the bridge gateway (172.18.0.1) instead of the site's real address, so the hole punch never validated (i.e., Site last hole punch is too old, registration skipped). After correcting the source-IP handling, the WG handshake still didn't complete: on wg show wg0 the peer had the right endpoint but no latest handshake, with only ~148 B-received / ~92 B-sent increments. This seems consistent with repeated WireGuard initiations and server responses (init 148 B, response 92 B) but not a working data plane. Switching this host to RootlessKit's pasta networking made handshake complete immediately (latest handshake: 2 seconds ago) and all resources recovered.

What worked for this environment (host-local; Debian 13, Docker 29 / RootlessKit 3.0):

• {"userland-proxy": false} in the rootless daemon.json. The host dockerd crash-looped until br_netfilter was also loaded with modprobe br_netfilter.
• Switched RootlessKit networking to pasta with DOCKERD_ROOTLESS_ROOTLESSKIT_NET=pasta + DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=implicit in the rootless docker systemd drop-in, then systemctl --user daemon-reload && systemctl --user restart docker.

Follow-Up to OP

The gerbil 1.4.2 dedup-cache-race hypothesis and the "second, gerbil-version-independent 1.20.0 bug" is probably wrong and a false conclusion.

Docs suggestion

Pangolin's supported rootless path is with Podman Quadlets, and it already requires Network=pasta (which is how I found my workaround), but rootless Docker has no documented equivalent that I could find. You can sort of work backwards from Podman to figure out what you need for docker, but of course that's nothing like the user-friendliness of a support path in the docs.

So I think the short answer here is: close this issue, root cause was an unobserved rootless Docker unattended upgrade that fried source IP handling.

[github-actions]

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.