Black map on cold start: /api/proxy-tiles 500s on style.json/sprite/tiles.json (UND_ERR_CONNECT_TIMEOUT) and MapLibre never retries

[Aiacos]

Summary

On a cold start (empty Next fetch cache), the first page load races the basemap requests against the app's own data-layer fetch storm. /api/proxy-tiles then returns 500 for style.json / sprite.* / tiles.json, and because MapLibre never retries a failed style, the map stays black for the whole browser session. A later reload works, which makes this look intermittent — but it reproduces 100% of the time from a cold cache.

This bites every self-hoster on first run, and again after every image update (a fresh container = empty /app/.next/cache/fetch-cache).

Environment

• Image: ghcr.io/simplifaisoul/osiris@sha256:92eeda3bdeda0d5292241ca25dfdbbd44e8ecb0f1212c06a57599949c64d73dc (rev c91e1ea, built 2026-07-30)
• App version: v4.1 — Next.js 16.2.6, Node 22.23.2
• Docker on Linux, bridge network, port 3005 → 3000, no proxy, no egress filtering

Symptoms

The UI shell renders fine — header, ticker, sidebars, entity counter all alive — but the map canvas is pure black.

Browser console:

Error: AJAXError: Internal Server Error (500):
  http://host:3005/api/proxy-tiles?url=https%3A%2F%2Fbasemaps.cartocdn.com%2Fgl%2Fdark-matter-gl-style%2Fstyle.json

Depending on timing it is style.json, sprite.json, sprite.png, or vector/carto.streets/v1/tiles.json that loses the race.

Server logs at the same moment:

Tile proxy error: TypeError: fetch failed
  [cause]: Error [ConnectTimeoutError]: Connect Timeout Error
           (attempted address: basemaps.cartocdn.com:443, timeout: 10000ms)
           code: 'UND_ERR_CONNECT_TIMEOUT'
Tile proxy error: TypeError: fetch failed
  [cause]: ConnectTimeoutError (attempted address: tiles-c.basemaps.cartocdn.com:443, timeout: 10000ms)
Tile proxy error: TypeError: fetch failed
  [cause]: ConnectTimeoutError (attempted address: tiles-d.basemaps.cartocdn.com:443, timeout: 10000ms)

The same connect timeouts hit unrelated upstreams in the same window, which is the giveaway that this is contention inside the Node process rather than an issue with Carto:

[OSIRIS] Taiwan THB fetch failed: ConnectTimeoutError (thbapp.thb.gov.tw:443, timeout: 10000ms)
[OSIRIS] GDACS fetch error:      ConnectTimeoutError (www.gdacs.org:443, timeout: 10000ms)
SatNOGS fetch error:             ConnectTimeoutError (db.satnogs.org:443, timeout: 10000ms)
NASA EONET normalization error:  AbortError: This operation was aborted

Reproduction (deterministic)

docker exec osiris rm -rf /app/.next/cache/fetch-cache
docker restart osiris
sleep 8
# open the dashboard immediately  ->  black map, 500 on proxy-tiles

Warm the six basemap assets once and the map renders on every subsequent load:

for u in \
  "https://basemaps.cartocdn.com/gl/dark-matter-gl-style/style.json" \
  "https://tiles.basemaps.cartocdn.com/gl/dark-matter-gl-style/sprite.json" \
  "https://tiles.basemaps.cartocdn.com/gl/dark-matter-gl-style/sprite.png" \
  "https://tiles.basemaps.cartocdn.com/gl/dark-matter-gl-style/sprite@2x.json" \
  "https://tiles.basemaps.cartocdn.com/gl/dark-matter-gl-style/sprite@2x.png" \
  "https://tiles.basemaps.cartocdn.com/vector/carto.streets/v1/tiles.json" ; do
  curl -s -o /dev/null -w "%{http_code} $u\n" \
    "http://localhost:3005/api/proxy-tiles?url=$(python3 -c "import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1],safe=''))" "$u")"
done

What I ruled out

This is not DNS, not egress filtering, and not Carto rate-limiting:

• dns.promises.lookup() from inside the container resolves basemaps.cartocdn.com and tiles.basemaps.cartocdn.com instantly.
• 50 concurrent raw net.connect(443, ...) from inside the container: 50/50 ok in 340 ms.
• 50 concurrent plain fetch() (bypassing the route) from inside the container: 50/50 ok in 487 ms.
• Each failing URL, requested on its own through /api/proxy-tiles, returns 200 in ~100 ms.

It only fails when the proxy requests are in flight at the same time as the app's own cold-start data fetches. Those are heavy — measured on this instance:

Endpoint	Time	Payload
/api/fires	17.3 s	258 KB
/api/cctv	7.1 s	3.37 MB
/api/satellites	0.08 s	2.35 MB
/api/flights	—	2.5 MB (Failed to set Next.js data cache … items over 2MB can not be cached)

Likely mechanism

src/app/api/proxy-tiles/route.ts does the upstream fetch through the Next data cache:

const res = await fetch(url.toString(), {
  headers: { Accept: '*/*', 'User-Agent': 'Osiris-Tile-Proxy/1.0' },
  next: { revalidate: 31536000 },
});

Two consequences:

1. Every failure is a hard 500 with no retry. catch → NextResponse.json({error:'Internal server error'}, {status:500}). One 10 s hiccup on style.json and MapLibre is done — it does not re-request a failed style, so the map is black until the user manually reloads.
2. Cache misses land in the libuv threadpool. On a cold cache, ~100 tile responses plus multi-MB API responses are being gzip-inflated (zlib → threadpool) and written to fetch-cache on disk (fs → threadpool) while undici's connect path needs dns.lookup() — which is also threadpool work, and only 4 threads by default. Queued getaddrinfo past 10 s presents exactly as UND_ERR_CONNECT_TIMEOUT even though the network is idle. That matches every observation above: fine in isolation, fine at 50 concurrent raw sockets, broken only when the cache-write/inflate storm is running.

After the assets are cached (revalidate: 31536000), the proxy answers in ~2 ms from disk and the problem disappears — until the next fresh container.

Suggested fixes

Roughly in order of effort:

1. Retry + don't fail hard. Wrap the upstream fetch in 2–3 retries with backoff, and give the connect a longer budget than undici's default 10 s for style/sprite/tiles.json specifically. These six requests are the difference between a working map and a black screen.
2. Warm the basemap assets at boot (server-side, on first request or via instrumentation.ts), so the style path is never a cold miss racing the data layers.
3. Stagger the cold-start data fetches. The layers do not all need to fire in the same tick; a small concurrency cap on the ingestion side would take the pressure off everything else, including the >2MB payloads that can't be cached anyway.
4. Consider not proxying the immutable Carto assets at all. The response already sets Cache-Control: public, max-age=31536000, immutable and Access-Control-Allow-Origin: * — for the vanilla Carto basemap the browser can fetch it directly (which is what the older OsirisMap.tsx did), keeping the server out of the critical path for rendering.
5. Document that /app/.next/cache is worth persisting in DOCKER.md — a volume there makes image updates non-disruptive.

(1) and (2) alone would fix the black map for self-hosters. Happy to send a PR for the retry + warm-up if that direction sounds right.

[Aiacos]

Two follow-ups after reading the repo properly rather than just the shipped bundle.

1. The repo already ships a tile cache — the map just doesn't use it

nginx/nginx.conf defines exactly the layer this issue is asking for:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=tile_cache:100m max_size=10g inactive=365d use_temp_path=off;

location ~ ^/proxy/tiles/(?<target_domain>(?:[a-d]\.)?basemaps\.cartocdn\.com)/(?<target_path>.*)$ {
    proxy_pass https://$target_domain/$target_path$is_args$args;
    proxy_cache tile_cache;
    proxy_cache_valid 200 304 365d;
    proxy_cache_valid any 1m;
    proxy_cache_lock on;              # Prevent thundering herd
}

proxy_cache_lock on plus a 10 GB disk cache is precisely the right answer to the cold-start stampede. But the client never requests that path — OsirisMap.tsx:160 rewrites every basemap URL to the Next route instead:

return { url: `${baseUrl}/api/proxy-tiles?url=${encodeURIComponent(url)}` };

So basemap traffic goes through location / → osiris:3000 → /api/proxy-tiles → undici, and the nginx tile_cache zone stays empty regardless of whether osiris-cache is running. Pointing the map at /proxy/tiles/<host>/<path> when the cache front-end is present would fix this issue with configuration that already exists in the tree, and would take the Node process out of the tile path entirely.

Worth noting the two are not equivalent on failure either: nginx caches failures for 1 minute and retries, whereas the Next route turns one 10 s connect timeout into a hard 500 that MapLibre never retries.

Two smaller things in the same area: nginx/nginx.conf and intel/ are not mentioned in DOCKER.md or README.md, so a self-hoster following the docs never learns the cache tier exists; and /api/entity/expand silently returns empty nodes/links when osiris-intel isn't running, which reads as "the feature is broken" rather than "a service is missing".

2. If you recommend persisting .next/cache, mention the ownership trap

I suggested this in the original post and hit the trap myself. The image runs as uid 1001 (nextjs), but Docker initialises a named volume as root:root when the mount point doesn't exist in the image:

$ docker exec osiris ls -ld /app/.next/cache
drwxr-xr-x 2 root root 4096 /app/.next/cache
$ docker exec osiris touch /app/.next/cache/_probe
touch: /app/.next/cache/_probe: Permission denied

Next swallows the write error, so there is no log line and no visible failure — the fetch cache is simply never populated, and every page load becomes a cold load. That turns an occasional black map into a permanent one. It needs a chown 1001 init step (or a pre-chowned bind mount) to actually help:

services:
  cache-init:
    image: busybox
    command: ["sh","-c","chown -R 1001:65533 /cache"]
    volumes: [next-cache:/app/.next/cache]
    restart: "no"

With the volume writable and a small sidecar priming the six style/sprite/tiles URLs after /api/health goes green, the map survives container restarts and Watchtower updates here. That's a workaround at the deployment layer though — the retry-and-don't-500 fix in the route is still the thing that would help everyone else.