Existing HTTP/3 connections become unusable during config update

[timmclean]

When a new config is pushed to caddy using POST http://localhost:2019, clients with existing HTTP/3 connections to the caddy server lose the ability to connect for a short period of time, causing active users to experience downtime.

I was able to reproduce this with a clean install on a new EC2 instance:

1. Create a t4g.nano instance on EC2 with Ubuntu 24.04 (arm64).
2. Open TCP ports 22, 80, 443 and UDP port 443.
3. Install Caddy v2.9.1 for Linux arm64
4. Create a config.json file on the server (replace TESTDOMAIN with a domain you control):

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [ ":443" ],
          "routes": [
            {
              "match": [ { "host": [ "TESTDOMAIN" ] } ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "group": "group1",
                      "handle": [
                        {
                          "body": "hello world",
                          "handler": "static_response"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [ "TESTDOMAIN" ]
          }
        ]
      }
    }
  }
}

5. Add an A record for TESTDOMAIN that matches the public IPv4 address of the EC2 instance.
6. Run curl -d @config.json -H 'content-type: application/json' http://localhost:2019/load on the server
7. Go to https://TESTDOMAIN in Google Chrome (tested with both Chrome for Android and desktop).
8. "hello world" should be shown.
9. Refresh the page a bunch. It should load just fine.
10. Edit config.json, changing "hello world" to "hello world 2".
11. Run sleep 3 && curl -d @config.json -H 'content-type: application/json' http://localhost:2019/load on the server
12. Quickly switch to Google Chrome and start refreshing the page rapidly. The refreshes will work until sleep 3 finishes, at which point the page will hang and refuse to load.
13. Navigate to https://TESTDOMAIN from a different browser and observe that the page does in fact load.
14. Refresh Google Chrome again, and observe that the page still does not load.
15. Wait 1-2 minutes.
16. Refresh Google Chrome again. The page will now load correctly.

Blocking UDP port 443 seems to fix this problem.

Log output during config change:

{"level":"info","ts":1742505571.2683778,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"40602","headers":{"Accept":["*/*"],"Content-Length":["885"],"Content-Type":["application/json"],"User-Agent":["curl/8.5.0"]}}
{"level":"info","ts":1742505571.2690651,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1742505571.269525,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1742505571.269545,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1742505571.2697465,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"warn","ts":1742505571.2697577,"msg":"quic listener tls configs are more than 2","number of configs":3}
{"level":"info","ts":1742505571.2697651,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1742505571.269796,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1742505571.2698002,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1742505571.269803,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1742505571.2698061,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["caddybugtest.timmclean.net"]}
{"level":"info","ts":1742505571.2698162,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1742505571.2710686,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
{"level":"info","ts":1742505571.2711222,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1742505571.2733417,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}

[mholt]

Thanks for the instructions; I will give it a try soon... in the meantime, does this only happen with Chrome?

(What about curl with http/3, or Firefox?)

[timmclean]

I just tried Firefox and the behaviour there is a bit better. When the config update goes through, it seems to hesitate for a moment and then fallback to HTTP/2 transparently. When I check the requests in the Network tab, I can see that they are HTTP/3 before the config change, and then HTTP/2 after the config change. I'm guessing the hesitation is because Firefox is setting up a new connection.

I haven't observed any issues with curl --http3-only so far in my testing. It recreates connections every time, so that makes sense to me.

Client versions (desktop):

• Chrome 134.0.6998.117
• Firefox 136.0.2
• curl 8.12.1 musl

[crrodriguez]

Chrome stable http3 support is spotty.. I have achieved better results with chrome-dev where it seems to be more consistent/stable.