Skip to content

Commit 7c65f6a

Browse files
MegaRedHandMegaRedHand
committed
feat(devnet-skill): surface each node's image commit in Grafana
Answering "which build is this node running" meant ssh-ing to the host for a docker inspect: the image tag is mutable, and only ethlambda embeds its SHA in lean_node_info (partner clients report a release string, e.g. lantern v0.0.5). Docker copies an image's OCI labels onto every container created from it and cAdvisor re-exports them as container_label_*, so revision/ref.name give a uniform cross-client answer that is already in the central Prometheus, with no new exporter or client-side change. Add an "Image build (commit per node)" table to both cAdvisor-backed dashboards. It reports the image the container was CREATED with, so it stays honest after a docker pull that was not followed by a recreate. On the resources dashboard it sits next to restart events and uptime, where a fresh uptime beside an unchanged commit reads as "came back on the same build". On the client dashboard the node is matched through cAdvisor's `name`, since cAdvisor's own `job` is `cadvisor` and its `instance` is the host, so neither of the other panels' filters applies; a `.+_[0-9]+` guard is ANDed on so the "All" nodes selection does not drag in the observability containers. The resources dashboard needs no guard: its `name` variable already carries that allValue.
1 parent ed86382 commit 7c65f6a

4 files changed

Lines changed: 258 additions & 2 deletions

File tree

‎.claude/skills/multi-server-devnet/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -456,9 +456,9 @@ pick their datasource through a template variable, so no editing):
456456

457457
| File | uid | Content |
458458
|---|---|---|
459-
| `client-dashboard.json` | `lean-ethereum-clients-dashboard` | The main per-node dashboard: Overview (start time, validators, committees, head/justified/finalized, slot + finality-delay graphs) plus 14 collapsed sections (config, sync, peers, req/resp + mesh, gossip, gossip arrival timing, PQ signatures, aggregation coverage, block production, proposal internals, fork choice, attestations, state transition, storage + tick health). Filters `network`/`job`/`instance` — `instance` is the **host**, not the node |
459+
| `client-dashboard.json` | `lean-ethereum-clients-dashboard` | The main per-node dashboard: Overview (start time, validators, committees, head/justified/finalized, slot + finality-delay graphs) plus 14 collapsed sections (config, sync, peers, req/resp + mesh, gossip, gossip arrival timing, PQ signatures, aggregation coverage, block production, proposal internals, fork choice, attestations, state transition, storage + tick health). Filters `network`/`job`/`instance` — `instance` is the **host**, not the node. The config section's **Image build** table answers "which commit is this node running" for **every** client, from the image's OCI `revision`/`ref.name` labels re-exported by cAdvisor as `container_label_*` (see `references/operations.md`) — ethlambda's own `lean_node_info{version}` already embeds its short SHA, partner clients' usually do not |
460460
| `finality-dashboard.json` | `devnet-finality-overview` | head / justified / finalized per devnet, one series per `network` |
461-
| `resources-dashboard.json` | `devnet-resources` | Per-node CPU cores + memory working set (+ limit + %-of-limit for OOM watch) from cAdvisor |
461+
| `resources-dashboard.json` | `devnet-resources` | Per-node CPU cores + memory working set (+ limit + %-of-limit for OOM watch), OOM kills, restart events, current uptime, and the same **Image build** commit-per-node table as the client dashboard — all from cAdvisor, so read a fresh uptime next to an unchanged commit as "came back on the same build" |
462462
| `logs-dashboard.json` | `devnet-logs` | Loki logs panel + log-volume-by-level, filtered `network`/`node`/`stream`/`search` |
463463
| `grafana-finality-alert.yaml.template` | — | Unified-alerting provisioning for the per-devnet "lost finality" rule + Slack contact point; placeholders filled by the deploy script |
464464

‎.claude/skills/multi-server-devnet/references/operations.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,27 @@ dashboard can safely use `.*` because promtail only ships
200200
emits the per-host file; `scripts/start-observability.sh` creates the per-host
201201
`prometheus` + `cadvisor` containers (it refuses to touch existing ones without
202202
`RECREATE=1`, since a live scraper's flags may not be the ones it would use).
203+
204+
**cAdvisor also carries the image's OCI labels**, which is the only cross-client
205+
way to see *which commit a node is actually running*: docker copies an image's
206+
`LABEL`s onto every container created from it, and cAdvisor (default
207+
`--store_container_labels=true`) re-exports them on every `container_*` series as
208+
`container_label_<label_with_underscores>`. So
209+
`container_label_org_opencontainers_image_revision` = the full 40-char SHA and
210+
`..._ref_name` = the branch/tag — for partner clients too, whose own
211+
`lean_node_info{version=…}` is often just a release string (`lantern` reports
212+
`v0.0.5`) while ethlambda's already embeds the short SHA. It reflects the image
213+
the container was **created** with, so it stays honest after a `docker pull` that
214+
was not followed by a recreate — unlike the mutable tag in `image`. The client
215+
dashboard's "Image build" table consumes it; if `Commit` is empty for a client,
216+
that client's Dockerfile simply does not set the label (ethlambda's does, fed by
217+
`GIT_COMMIT`/`GIT_BRANCH` build args from `docker_publish.yaml`). Two matcher
218+
gotchas when writing such a panel: cAdvisor's `job` is `cadvisor` and its
219+
`instance` is the **host**, so neither the `$job` nor the `$instance` filter of
220+
the node panels applies — match the node through cAdvisor's `name` instead
221+
(`name=~"$job"`, since the container name IS the node's job label), and AND a
222+
`name=~".+_[0-9]+"` guard onto it so "All" (`.*`) does not drag the infra
223+
containers in, per the gotcha above.
203224
The **central** stack is a separate one-time deployment: a prometheus started with
204225
`--web.enable-remote-write-receiver` on a routable address + distinct port (e.g.
205226
`:9099`), a Loki with `allow_structured_metadata: true` + schema v13/tsdb and

‎.claude/skills/multi-server-devnet/scripts/client-dashboard.json‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1677,6 +1677,184 @@
16771677
}
16781678
}
16791679
]
1680+
},
1681+
{
1682+
"datasource": {
1683+
"type": "prometheus",
1684+
"uid": "${datasource}"
1685+
},
1686+
"description": "Which build each node is actually running, read from the OCI labels baked into its container image (`org.opencontainers.image.revision` / `.ref.name`) and exported by cAdvisor as `container_label_*`. This is the label of the image the container was CREATED with, so it stays correct after a `docker pull` that has not been followed by a recreate -- unlike the image tag, which is mutable.\n\nCoverage depends on the client's Dockerfile setting the labels; a client that does not leaves Commit empty. Filtered by $network and $job only: cAdvisor's `instance` is the HOST and its `job` is `cadvisor`, so neither the $job nor the $instance matcher of the other panels applies -- the node is matched through cAdvisor's `name` (the container name, same string as the node's `job`).",
1687+
"fieldConfig": {
1688+
"defaults": {
1689+
"custom": {
1690+
"align": "auto",
1691+
"cellOptions": {
1692+
"type": "auto"
1693+
},
1694+
"filterable": true,
1695+
"footer": {
1696+
"reducers": []
1697+
},
1698+
"inspect": false
1699+
},
1700+
"mappings": [],
1701+
"noValue": "No cAdvisor data for this network -- is its cadvisor container scraped?"
1702+
},
1703+
"overrides": [
1704+
{
1705+
"matcher": {
1706+
"id": "byName",
1707+
"options": "Node"
1708+
},
1709+
"properties": [
1710+
{
1711+
"id": "custom.width",
1712+
"value": 130
1713+
}
1714+
]
1715+
},
1716+
{
1717+
"matcher": {
1718+
"id": "byName",
1719+
"options": "Client"
1720+
},
1721+
"properties": [
1722+
{
1723+
"id": "custom.width",
1724+
"value": 110
1725+
}
1726+
]
1727+
},
1728+
{
1729+
"matcher": {
1730+
"id": "byName",
1731+
"options": "Commit"
1732+
},
1733+
"properties": [
1734+
{
1735+
"id": "custom.width",
1736+
"value": 350
1737+
}
1738+
]
1739+
},
1740+
{
1741+
"matcher": {
1742+
"id": "byName",
1743+
"options": "Branch / tag"
1744+
},
1745+
"properties": [
1746+
{
1747+
"id": "custom.width",
1748+
"value": 230
1749+
}
1750+
]
1751+
},
1752+
{
1753+
"matcher": {
1754+
"id": "byName",
1755+
"options": "Image built"
1756+
},
1757+
"properties": [
1758+
{
1759+
"id": "custom.width",
1760+
"value": 230
1761+
}
1762+
]
1763+
}
1764+
]
1765+
},
1766+
"gridPos": {
1767+
"h": 9,
1768+
"w": 24,
1769+
"x": 0,
1770+
"y": 47
1771+
},
1772+
"id": 215,
1773+
"options": {
1774+
"cellHeight": "sm",
1775+
"showHeader": true,
1776+
"sortBy": [
1777+
{
1778+
"desc": false,
1779+
"displayName": "Node"
1780+
}
1781+
]
1782+
},
1783+
"pluginVersion": "12.3.2",
1784+
"targets": [
1785+
{
1786+
"datasource": {
1787+
"type": "prometheus",
1788+
"uid": "${datasource}"
1789+
},
1790+
"editorMode": "code",
1791+
"exemplar": false,
1792+
"expr": "last_over_time(container_last_seen{network=~\"$network\", name=~\"$job\", name=~\".+_[0-9]+\"}[5m])",
1793+
"instant": true,
1794+
"legendFormat": "__auto",
1795+
"range": false,
1796+
"refId": "A"
1797+
}
1798+
],
1799+
"title": "Image build (commit per node)",
1800+
"transformations": [
1801+
{
1802+
"id": "labelsToFields",
1803+
"options": {
1804+
"keepLabels": [
1805+
"name",
1806+
"client_type",
1807+
"container_label_org_opencontainers_image_revision",
1808+
"container_label_org_opencontainers_image_ref_name",
1809+
"image",
1810+
"container_label_org_opencontainers_image_created"
1811+
]
1812+
}
1813+
},
1814+
{
1815+
"id": "merge",
1816+
"options": {}
1817+
},
1818+
{
1819+
"id": "filterFieldsByName",
1820+
"options": {
1821+
"include": {
1822+
"names": [
1823+
"name",
1824+
"client_type",
1825+
"container_label_org_opencontainers_image_revision",
1826+
"container_label_org_opencontainers_image_ref_name",
1827+
"image",
1828+
"container_label_org_opencontainers_image_created"
1829+
]
1830+
}
1831+
}
1832+
},
1833+
{
1834+
"id": "organize",
1835+
"options": {
1836+
"excludeByName": {},
1837+
"includeByName": {},
1838+
"indexByName": {
1839+
"name": 0,
1840+
"client_type": 1,
1841+
"container_label_org_opencontainers_image_revision": 2,
1842+
"container_label_org_opencontainers_image_ref_name": 3,
1843+
"image": 4,
1844+
"container_label_org_opencontainers_image_created": 5
1845+
},
1846+
"renameByName": {
1847+
"name": "Node",
1848+
"client_type": "Client",
1849+
"container_label_org_opencontainers_image_revision": "Commit",
1850+
"container_label_org_opencontainers_image_ref_name": "Branch / tag",
1851+
"image": "Image",
1852+
"container_label_org_opencontainers_image_created": "Image built"
1853+
}
1854+
}
1855+
}
1856+
],
1857+
"type": "table"
16801858
}
16811859
],
16821860
"title": "Devnet configuration",

‎.claude/skills/multi-server-devnet/scripts/resources-dashboard.json‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,63 @@
310310
"expr": "time() - max by (name) (container_start_time_seconds{network=\"$network\", name=~\"${name:pipe}\"})"
311311
}
312312
]
313+
},
314+
{
315+
"id": 8,
316+
"type": "table",
317+
"title": "Image build (commit per node) — $network",
318+
"description": "Which build each node is actually running, read from the OCI labels baked into its container image (org.opencontainers.image.revision / .ref.name): docker copies an image's labels onto every container created from it, and cAdvisor re-exports them as container_label_*. It is the label of the image the container was CREATED with, so it stays correct after a `docker pull` that was not followed by a recreate, unlike the mutable tag in Image. An empty Commit means that client's Dockerfile does not set the label. Read it next to 'Restart events' / 'Current uptime': a fresh uptime with an unchanged Commit means the node came back on the SAME build, a changed one means the restart also upgraded it.",
319+
"datasource": { "type": "prometheus", "uid": "${ds_prom}" },
320+
"gridPos": { "h": 9, "w": 24, "x": 0, "y": 40 },
321+
"fieldConfig": {
322+
"defaults": {
323+
"custom": { "align": "auto", "cellOptions": { "type": "auto" }, "filterable": true, "inspect": false },
324+
"noValue": "No cAdvisor data for this network. Is its cadvisor container being scraped?"
325+
},
326+
"overrides": [
327+
{ "matcher": { "id": "byName", "options": "Node" }, "properties": [{ "id": "custom.width", "value": 130 }] },
328+
{ "matcher": { "id": "byName", "options": "Client" }, "properties": [{ "id": "custom.width", "value": 110 }] },
329+
{ "matcher": { "id": "byName", "options": "Commit" }, "properties": [{ "id": "custom.width", "value": 350 }] },
330+
{ "matcher": { "id": "byName", "options": "Branch / tag" }, "properties": [{ "id": "custom.width", "value": 230 }] },
331+
{ "matcher": { "id": "byName", "options": "Image built" }, "properties": [{ "id": "custom.width", "value": 230 }] }
332+
]
333+
},
334+
"options": {
335+
"cellHeight": "sm",
336+
"showHeader": true,
337+
"sortBy": [{ "desc": false, "displayName": "Node" }]
338+
},
339+
"targets": [
340+
{
341+
"refId": "A",
342+
"datasource": { "type": "prometheus", "uid": "${ds_prom}" },
343+
"editorMode": "code",
344+
"instant": true,
345+
"legendFormat": "__auto",
346+
"expr": "last_over_time(container_last_seen{network=\"$network\", name=~\"${name:pipe}\"}[5m])"
347+
}
348+
],
349+
"transformations": [
350+
{ "id": "labelsToFields", "options": { "keepLabels": ["name", "client_type", "container_label_org_opencontainers_image_revision", "container_label_org_opencontainers_image_ref_name", "image", "container_label_org_opencontainers_image_created"] } },
351+
{ "id": "merge", "options": {} },
352+
{ "id": "filterFieldsByName", "options": { "include": { "names": ["name", "client_type", "container_label_org_opencontainers_image_revision", "container_label_org_opencontainers_image_ref_name", "image", "container_label_org_opencontainers_image_created"] } } },
353+
{
354+
"id": "organize",
355+
"options": {
356+
"excludeByName": {},
357+
"includeByName": {},
358+
"indexByName": { "name": 0, "client_type": 1, "container_label_org_opencontainers_image_revision": 2, "container_label_org_opencontainers_image_ref_name": 3, "image": 4, "container_label_org_opencontainers_image_created": 5 },
359+
"renameByName": {
360+
"name": "Node",
361+
"client_type": "Client",
362+
"container_label_org_opencontainers_image_revision": "Commit",
363+
"container_label_org_opencontainers_image_ref_name": "Branch / tag",
364+
"image": "Image",
365+
"container_label_org_opencontainers_image_created": "Image built"
366+
}
367+
}
368+
}
369+
]
313370
}
314371
]
315372
}

0 commit comments

Comments
 (0)