obs(deploy-probe): NR alert + Prom rule + dashboard tiles + catalog

[mastermanas805]

Summary

Companion to worker PR InstaNode-dev/worker#69 (hourly synthetic deploy prober). Surfaces the new prober metric on all four operator surfaces in one PR per CLAUDE.md rule 25.

• newrelic/alerts/deploy-probe-fail.json — P0 NRQL alert: ANY instant_deploy_probe_outcome_total{result="fail"} over a 30-minute window. 30m guarantees a P0 inside one tick at the 60-min cadence rather than waiting two cycles (90 minutes of broken deploys before paging would be too long).
• k8s/prometheus-rules.yaml — Prom DeployProbeFail rule under the new instant-worker-deploy-probe group, mirrors the alert.
• newrelic/dashboards/instanode-reliability.json — three tiles matching the AUTH-004 layout (row 28): outcomes-per-leg line chart (6h), fails billboard (must-be-0), P95 latency line per leg.
• observability/METRICS-CATALOG.md — two rows (instant_deploy_probe_outcome_total + instant_deploy_probe_latency_seconds), both marked lazy with the AUTH-004-style operator note about first-tick materialisation.

Coverage block

Symptom:        hourly deploy prober metric exists in worker but no
                alert / dashboard / catalog entry (rule 25 violation)
Enumeration:    grep -n "deploy_probe\|deploy-probe" k8s/prometheus-rules.yaml
                observability/METRICS-CATALOG.md
                newrelic/dashboards/instanode-reliability.json
                newrelic/alerts/
Sites found:    4 (3 existing files + 1 new alert)
Sites touched:  4
Live verified:  awaiting operator apply (kubectl apply -f
                k8s/prometheus-rules.yaml + nrcli sync the new alert +
                dashboard re-import).

Test plan

• python3 -c "import json; json.load(open('newrelic/alerts/deploy-probe-fail.json'))" → valid JSON
• python3 -c "import json; json.load(open('newrelic/dashboards/instanode-reliability.json'))" → still valid JSON after edit
• python3 -c "import yaml; yaml.safe_load(open('k8s/prometheus-rules.yaml'))" → still valid YAML after edit
• Alert JSON keys match auth-probe-fail.json shape exactly (same shape contract)
• Operator kubectl apply -f k8s/prometheus-rules.yaml -n monitoring after worker PR infra(data-tier): S4 razorpay sig-fail monitoring + R6 nats PVC + R7 eviction protection + apply runbook #69 ships and prod starts emitting the metric.
• Operator import deploy-probe-fail.json to NR (via NerdGraph or the alert-sync script).
• Operator re-import the dashboard JSON to NR.
• Confirm the alert fires on a test breakage (e.g. temporarily point DEPLOY_PROBE_BEARER_TOKEN at a bad value and verify P0 paged within 30 min).

🤖 Generated with Claude Code