Testnet observability: metrics, cold-path probe, health workflow - #436
Merged
Merged
Conversation
/metrics served only the Go runtime. The control plane is the one place that knows the mesh's size and state -- enrolled nodes, active router leases and what each router reported attached to it -- yet none of that was observable without an admin token and a JSON parser. A store-backed collector now exports it as gauges, refreshed at most every 30s so scrapers cannot hammer the database. The headline figure is sam_control_plane_mesh_connected_peers: distinct non-router peers attached to at least one active router, i.e. how many nodes are really on the mesh right now, as opposed to how many rows are in the table. A failed refresh keeps the last snapshot and flips a scrape_success gauge so a store outage is visible without the figures vanishing. Every registered route is also counted and timed, labelled by mux pattern rather than request path so peer and token ids never become labels. Probes and /metrics itself are left uncounted. The collector lives in a per-server registry merged with the default one at serve time: tests and sam-one build several servers per process, and a global registration would panic on the second.
The router had no HTTP listener at all: no probes, and libp2p's own Prometheus metrics (relay reservations, relayed bytes, connection churn, rcmgr limits) were collected into the default registry with nowhere to be read from. --metrics-addr opens a plain listener carrying those plus the state only the router has: peers that completed the mesh handshake as opposed to merely holding a transport connection, the ban list size, DHT routing table size and the router's own credential expiry, with counters for handshake and lease-renewal outcomes. Off by default and never shared with the libp2p ports, since nothing on it is authenticated. The listener comes up before enrollment so /healthz answers while the control plane is still being reached; /readyz turns 200 only once the host is online, which is what a kubelet should wait for. The collector reads Host and DHT only after observing isReady, which is stored after they are assigned.
GET /debug/mesh-info and /debug/token-info answer the useful questions -- am I on the mesh, how many peers, when does my credential expire -- but only on demand, to a caller holding the API token. As gauges a scraper keeps, every node that runs unattended becomes a continuous probe of the mesh; the testnet canaries are exactly that. The sidecar's /metrics now carries them (still token-gated on TCP), and --metrics-addr opens a separate unauthenticated listener with /metrics, /healthz and /readyz. That listener is the only way a socket-only node, which has no TCP port at all, can be scraped or probed by a kubelet, and it means neither ever needs the API token. /readyz is 503 until the node holds an authenticated connection to a router. The collector is per node, in a registry merged with the default one at serve time, for the same reason as the control plane's: several nodes share a process in tests and sam-one.
…rometheus The testnets ran blind: nothing looked at a component after `kubectl rollout status` returned. GKE already runs Managed Prometheus, so a PodMonitoring per target is all it takes to start keeping the metrics the previous commits export. The router StatefulSet gains a cluster-internal metrics port (no hostPort) and, for the first time, real probes: startup and liveness on /healthz, readiness on /readyz. Every canary node gets --metrics-addr and a named metrics port, plus a shared sam-canary=true label that one PodMonitoring in the canary namespace selects; the box canary also exposes sam-box's flow counters. The sam-canary node uses 9091 because the agnhost container in that same pod already owns 9090. deploy.yaml applies the monitoring template only when the PodMonitoring CRD exists, so a cluster without managed collection still rolls out. The canary-manifest e2e test learns that PodMonitoring is one of the kinds a kind cluster is allowed to lack.
sam-mesh's router had only a tcpSocket probe and no scrapeable port; sam-node had no probes at all, its API being loopback-only. Both now serve --metrics-addr on a named metrics containerPort (never a hostPort or Service, since it is unauthenticated) and probe it: startup and liveness on /healthz, readiness on /readyz, so "ready" means enrolled and on the mesh rather than "a socket is open". Scraping is opt-in per collector because each resource is a CRD the cluster may not have: monitoring.podMonitoring.enabled renders a Google Managed Prometheus PodMonitoring per component, monitoring.podMonitor the prometheus-operator PodMonitor. sam-node's metricsPort: 0 disables the listener, probes and scrape resources together.
b0c5c3c deleted site/content/docs/snippets/banana_bot_playground.py, the script deploy.yaml cats into the cop canary's ConfigMap. cat failing left the variable empty, python exited 0 on an empty file, and the pod has been in CrashLoopBackOff on both testnets since -- unnoticed, because the deploy step's rollout check passed on a pod with no readiness probe. The other canaries cover the node path, so the canary goes rather than the script coming back. The live Deployments were deleted by hand.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces Prometheus metrics scraping, health/readiness probes, and a background node garbage collection loop across the SAM mesh components, alongside a cold-path connectivity probe cronjob and updated Helm charts. The review feedback highlights critical safety improvements in the newly added Go code, specifically pointing out potential nil pointer dereferences in the control plane's garbage collection logic (node_gc.go) and the node's metrics collector (node_metrics.go) that could lead to runtime panics.
Every pod restart without a persistent data dir enrolls a fresh identity and nothing ever removed the old row: bananas holds 2615 enrolled nodes for about 17 peers actually attached to a router. Once a node's OIDC session lapses the row is dead weight -- /refresh refuses it whatever its autonomous-recovery setting, and a re-enrollment simply upserts a new one -- yet ListNodes walks it on every /admin/status and metrics refresh. An hourly sweep deletes unbanned nodes whose session expired more than --node-retention (default 30 days) ago, and counts what it removed. A ban is the row, so banned nodes are never touched, nor are nodes with no session bound. Replicas each sweep the shared database; the delete is idempotent, so no claim is needed.
Its one service, dummy-http, was an MCP-typed entry pointing at agnhost, which does not speak MCP. Since the node began probing backends before advertising them it has withheld the service on every start, so the three pods have contributed nothing to the mesh but rows in the nodes table; a discover on bananas today returns everything, vllm-tpu and openrouter, and no dummy-http. The realistic canaries cover what it was for. The shared sam-node-sa lived in that template; it moves to its own, applied with the namespaces so every canary finds it. The live Deployments were deleted by hand.
The canaries enroll once and stay; between rollouts nothing re-checks that a node with no identity can still join. A CronJob does: sam-node runs as a native sidecar with an empty data dir, and a curl container waits for its /readyz (enrolled and authenticated to a router), pings the router, discovers the everything canary's MCP server -- retrying while a just-joined node's routing table fills -- and sends it an MCP initialize through the mesh. One JSON line per run carries the timings for a log-based metric; the Job's own status is the verdict. deploy.yaml runs it once after the last canary and fails the deploy if it does not pass, which is the check the cop canary's weeks-long crash loop would have needed. Each run enrolls a fresh identity that the control plane's retention sweep later reclaims.
Nothing watched a testnet once deploy.yaml returned; the cop canary sat in CrashLoopBackOff for weeks with a green deploy behind it. This scheduled workflow asks, per testnet, whether a new user could join right now: the public /info answers and names routers, the control plane reports leases held and peers attached (read through the API server, since /metrics is deliberately not routed to the internet), routers and canaries are fully rolled out, and the cold-path probe succeeded within its last three schedules. The result lands where people already look: a table in the run summary, a badge on the README, and one "Testnet <env> is unhealthy" issue that is kept current rather than commented on every run and closes itself on the next green pass.
aojea
force-pushed
the
observability-metrics
branch
from
September 18, 2026 18:12
380af48 to
a40a969
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We had zero visibility on the testnets: nothing looked at a component after
kubectl rollout statusreturned, and the cop canary sat in CrashLoopBackOff on both testnets for weeks behind green deploys. This PR makes the mesh's size and state observable and puts a signal where people look.What lands
Metrics (no new dependencies)
/metricsnow exports mesh state from the store —sam_control_plane_mesh_connected_peers(distinct non-router peers attached to an active router: the real mesh size),routers_active, per-router connected peers / DHT size / lease age,enrolled_nodes{role,state}, enrollment requests, bootstrap tokens by state — plus per-route request counts and latencies. Cached 30s, per-server registry (tests and sam-one build several servers per process).--metrics-addrserving/metrics,/healthz,/readyz(off by default). Exposes libp2p's relay/connection metrics, which were already collected with nowhere to be read, plussam_router_authenticated_peers, handshake and lease-renewal outcome counters. The router had no HTTP listener and no probes before./debug/*values as gauges (sam_node_mesh_connected,connected_peers,biscuit_expiry,services_registered{type}) and the same--metrics-addr, which is the only way a socket-only node can be scraped or probed.Testnets (
.github/k8s,deploy.yaml)PodMonitoringfor control plane, routers and canaries — GKE Managed Prometheus is already running on the cluster, so scraping starts on apply. Router gets real HTTP probes.everythingMCP server →initializethrough the mesh → exit. Also run once per rollout; the deploy fails if a new user couldn't join. Validated in kind (ready 2s, initialize 12ms).dummy-httpwas MCP-typed over a non-MCP backend; the node correctly never advertised it — a live discover on bananas returnseverything,vllm-tpu,openrouter). Live Deployments were removed by hand;sam-node-samoved to its own template.--node-retention(default 30d after session expiry). bananas holds 2615 node rows for ~17 attached peers because every restart without a data dir enrolls a fresh identity and nothing ever deleted the old row. Banned rows are never touched.Helm charts: router and node metrics ports with startup/readiness/liveness probes; opt-in
monitoring.podMonitoring(GMP) /monitoring.podMonitor(prometheus-operator). 59/59 chart tests.GitHub-side signal:
testnet-health.yamlruns every 30 min per testnet — public/info, control-plane gauges via the API-server proxy, router/canary rollouts, probe freshness — writes a summary table, drives a README badge, and opens/updates/closes onetestnet-healthissue per testnet.Notes for the reviewer
scheduletrigger; if the WIF provider condition pinsworkflow_ref, that needs a one-line change on the GCP side.hubdeploys from the last release tag, so its probe and control-plane gauges appear only after the next release; until then the hub health job will report those two rows red.make lint's deadcode stage panics locally on go1.27.1 (pre-existing, x/tools SSA); passes with the go.mod toolchain. golangci clean, fullgo test -race ./...green,canary_manifests.batsgreen.