Hosted overlay is missing FountainBrokerCAInstallFailureRate (and three other portable rules)

[jhgaylor]

What is missing

FountainBrokerCAInstallFailureRate shipped in the portable rules (#1958), but
the hosted instance does not read that file. deploy/k8s/kustomization.yaml
keeps prometheusrule.yaml commented out, and
jhgaylor/home-cloud at platform/fountain-site/prometheusrule.yaml is a
hand-maintained copy of the rules rather than a reference to the portable
one. Until that copy gains the rule, the alert exists in the repository and
fires nowhere.

This matters more than usual here: the alert is ask 3 of #1671, and ask 1
(why CA installs regressed 0 -> 18% on 2026-09-06) is the one that needs the
signal in production to make progress.

The drift is not new

Diffing the two files today, three portable rules were already never copied
across:

• FountainReattachFailures
• FountainTurnFailureRate
• FountainTurnFirstOutputSlow

FountainBrokerCAInstallFailureRate becomes the fourth. The observability
guide already warns that "the hosted overlay requires its own rollout; a merge
of this file does not update it", so nothing is behaving unexpectedly; there is
just no gate that notices when the two drift apart.

Asks

1. Add FountainBrokerCAInstallFailureRate to
   platform/fountain-site/prometheusrule.yaml in jhgaylor/home-cloud and
   roll it out. Do this after the metric is actually being scraped in
   production, so the rule is not evaluating an absent series.
2. Decide whether the other three belong in the overlay, or whether they were
   deliberately left out.
3. Consider a check that diffs portable alert names against the overlay's, in
   whichever repo can see both. The docs table already has a name-parity test
   (Fountain.DocsTest, "the portable alert table matches the shipped rule
   names") and this is the same shape one level out.

Verifying

The metric only exists once the image carrying #1957 is running. Confirm the
series is present before adding the rule:

fountain_broker_ca_install_count{namespace="fountain"}

[jhgaylor]

Precondition already met. Production is running `sha-1a463d98` (the whole stack), and the series is scraping:

sum by (provider,outcome) (fountain_broker_ca_install_count)
=> {provider="sprites", outcome="ok"}  4

Four conversation CA setups since the rollout, all ok, no exit/unreachable/unavailable yet. So ask 1 of this issue can proceed whenever someone picks it up — the rule will not be evaluating an absent series.

Nothing to read into the clean run yet: the alert needs at least five attempts in an hour before it can fire, and #1671 saw its failures in bursts.