Repository navigation
Readiness stays 500 for 3h after a transient DB error during boot (TrackerScriptCache refresh_all never retried) #6647
Description
Past Issues Searched
Searched for TrackerScriptCache / tracker_script_cache / readiness; nothing matching.
Version
Community Edition v3.2.1 (also checked master: same code in lib/plausible/cache.ex and system_controller.ex).
What happened
A single transient DB error during boot makes /api/health (readiness) return 500 for 3 hours, while the app otherwise serves traffic fine. Under Kubernetes with the documented health probe this becomes an infinite restart loop.
Sequence, from logs:
- At boot
Plausible.Cache.Warmerrunsrefresh_allfor every cache at once, plus Oban starts. With a 10-connection pool and a DB a few hundred ms away, the pool hitDBConnection.ConnectionError ... queue_timeoutafter ~6s. Plausible.Cache.refresh/3catches that and only logsError refreshing 'tracker_script_cache' - ...(cache.ex ~L233). The cache stays empty.TrackerScriptCacheis in@critical_cachesfor readiness.ready?/1returnsfalsebecausesize() == 0butcount_all() == 9. Readiness responds500 {"sessions":"ok","postgres":"ok","clickhouse":"ok","sites_cache":null}.- The next
refresh_allfortracker_script_cacheis scheduled ~3 hours out (interval 10805730in the warmer init log).refresh_updated_recentlyruns every 30s but only picks up rows updated since the last run, so it never repairs a cache that missed the initial load. - If the deployment's startup/liveness probe points at
/api/health, the pod is killed long before 3h elapse, and the cycle repeats on every boot. I saw 56 consecutive restarts.
Plausible.Site.Cache and Plausible.Shield.IPRuleCache have the same shape but shorter refresh_all intervals (15 min / 3 min), so they self-heal before anyone notices.
Expected
A failed boot-time refresh_all should retry (short backoff) instead of waiting for the next multi-hour cycle, or refresh_updated_recently should fall back to a full refresh while ready?/1 is false.
Workaround
bin/plausible rpc 'Plausible.Site.TrackerScriptCache.refresh_all()' inside the container makes readiness green immediately. For Kubernetes, point startup/liveness at /api/system/health/live and keep /api/health for readiness only.
Activity
deepanshupal commentedon Sep 18, 2026
Confirmed on current
master(16120d0a). The three-hour failure window follows directly from the supervision setup:TrackerScriptCache.Allruns every 180 minutes, whileTrackerScriptCache.RecentlyUpdatedruns every 30 seconds but only queries rows updated in the last 15 minutes.Plausible.Cache.refresh/3catches the boot DB failure and always returns:ok, soPlausible.Cache.Warmercannot distinguish a failed initial fill from a successful cycle and schedules the normal interval.The smallest shared fix is probably to make cache refresh success observable to the warmer and retry a failed cycle with bounded backoff. Changing
refresh_updated_recentlyto callrefresh_allwheneverready?is false would self-heal this cache, but it also means the 30-second warmer can repeatedly issue a full-table query during a persistent DB incident and causesready?to run another source count query. A warmer-level retry avoids that and covers every cache with the same swallowed-failure shape, but it changes the existingrefresh_all/1 :: :okcontract.I can add a deterministic
Plausible.Cache.Warmerregression with an injected warmer function that fails once and records the retry timing, then implement a bounded retry result without needing Postgres timing. Before doing that: do maintainers prefer the shared warmer contract (:ok | {:error, reason}+ short retry) or a TrackerScriptCache-specific empty-cache fallback inrefresh_updated_recently/1?