fix: move runner configs to epoxy

[MasterPtato]

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[MasterPtato]

• fix(epoxy): use vbare #4616
• fix: dont upsert for ensure if missing #4615
• fix(envoy): require runner config for ws conn #4612
• fix(pb): backport serverless fix #4611
• fix(pb): optimize actor key allocation #4610
• fix: move runner configs to epoxy #4605 👈 (View in Graphite)
• chore(epoxy): clean up kv get #4604
• fix(pb-envoy): reduce round trips for ws connection #4603
• fix(envoy-client): fix handling actor event ack #4599
• fix(envoy-client): add tls #4595 : 1 other dependent PR (#4597 )
• chore: fix driver test suite #4591
• chore: tunnel auth #4588
• chore: remove udb as a dependency of envoy-client #4587
• chore: revert actor v2 hack #4586
• fix: fix pool metadata protocol version refresh, rivetkit native ws #4585
• feat: US-001 - Define SqliteKv trait in rivetkit-sqlite-native #4584
• feat(rust/engine-runner): minimal rust engine runner #4289
• chore: delete kv channel #4583
• feat(engine): rust runner #4582
• chore: remove non-engine drivers #4574 : 1 other dependent PR (#4581 )
• feat: dynamic actors #4397
• fix: driver test suite & sqlite reliability #4559
• fix: remove serverless token #4572
• fix: misc token fixes #4569
• fix(pb): clean up actor stop decision handling #4566
• fix(envoy): use global instance, add signal handlers #4565
• chore: misc fixes, add pb snapshot test #4561
• fix(cache): make in memory cache global #4560
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: fix: move runner configs to epoxy

Summary

This PR refactors how runner pool configurations are stored and discovered across datacenters. Instead of querying remote DCs over HTTP, configs are now written to epoxy (distributed KV) under a new GlobalDataKey encoding the originating DC label. A backfill workflow (runner_pool_backfill) migrates existing UDB-local configs. The PR also adds save_empty caching to the optimistic epoxy KV read path to avoid repeated fanout on negative cache misses.

---

Bug: GlobalDataKey::unpack Missing DATA Segment

File: engine/packages/pegboard/src/keys/runner_config.rs

TuplePack serializes 7 elements: (RUNNER, CONFIG, GLOBAL, DATA, dc_label, namespace_id, name), but TupleUnpack only destructures 6 elements: (usize, usize, usize, u16, Id, String). The DATA constant (value 4) is skipped.

This means:

• dc_label always decodes as 4 (the numeric value of DATA), not the actual dc_label
• namespace_id receives bytes from the actual dc_label field, likely a parse error or silently wrong UUID
• name receives bytes from namespace_id

Fix: add a usize placeholder for the DATA segment:

let (input, (_, _, _, _, dc_label, namespace_id, name)) =
    <(usize, usize, usize, usize, u16, Id, String)>::unpack(input, tuple_depth)?;

This would cause list_runner_config_enabled_dcs to malfunction and the backfill to produce entries that cannot be unpacked correctly.

---

Bug: ensure_normal_if_missing Now Overwrites Existing Configs

File: engine/packages/pegboard/src/ops/runner_config/ensure_normal_if_missing.rs

The function doc comment says "Creates a default normal runner config if one does not already exist", but the implementation now calls upsert unconditionally. This will overwrite a serverless config with a Normal {} config every time a runner2 workflow calls EnsureRunnerConfig. The original guard that skipped the write when a config already existed has been removed.

Combined with the existing // TODO: Race comment in upsert.rs, this race is now exercised on every new runner connection.

---

Issue: mutable Field Placeholder in Empty-Value Cache Write

File: engine/packages/epoxy/src/ops/kv/get_optimistic.rs

The // TODO: What should this be set to? on the mutable: true field in the empty-value CommittedValue should be resolved before merging. Even if semantically irrelevant today (empty-value returns None immediately in get_local.rs), leaving it unresolved is a maintenance hazard if sentinel behavior changes.

---

Issue: Race Guard Removed from cache_fanout_value

File: engine/packages/epoxy/src/ops/kv/get_optimistic.rs

The old implementation guarded against the race where a commit lands between the fanout read and the cache write by checking if a committed value with an equal or newer version already existed before writing. This guard was removed. If a committed value lands locally in that window, the cache will briefly serve a stale value. For mutable runner configs, this could silently serve an outdated config until TTL expiry. If this regression is intentional, it should be documented.

---

Issue: Missing epoxy Deletion in runner_config::delete

File: engine/packages/pegboard/src/ops/runner_config/delete.rs

Deleting a runner config removes it from local UDB but does not remove the corresponding GlobalDataKey entry from epoxy. list_runner_config_enabled_dcs will continue returning DC labels for deleted configs indefinitely. If tombstoning is handled separately, this should be documented.

---

Stale Log Message in Backfill Workflow

File: engine/packages/pegboard/src/workflows/runner_pool_backfill.rs

tracing::warn!("timed out processing pending actors metrics") appears to be copy-pasted from another backfill workflow. Should be something like "timed out reading runner pool configs".

---

Minor Issues

1. useless_format in upsert.rs - format!("invalid header name: too long (max 128)") has no format args. Should be a plain .to_string(). Clippy will flag these.

2. Duplicated protocol_version fetch in list.rs - Both arms of the if input.variant.is_some() check have identical fetch code. Could be deduplicated.

3. Backfill chunk timeout discards partial progress - When a backfill chunk times out at 2500ms, entries collected before the timeout are discarded. Resuming re-reads and re-writes the same range (idempotent), but progress within a timed-out chunk is entirely lost. Acceptable, but worth documenting.

4. DatacentersIntoIter placement - The new iterator type is declared between two IntoIterator impls rather than adjacent to DatacentersIter. Minor style inconsistency.

---

What Looks Good

• Structured logging with ? formatters throughout. No println!/eprintln! usage.
• Custom RivetError-derived errors follow the established pattern.
• Timestamp and key naming conventions followed.
• No Mutex<HashMap> usage; concurrent operations use buffer_unordered.
• The save_empty caching addition is a reasonable optimization for the negative-cache case.
• IntoIterator for DatacentersRepr follows the existing IntoIterator for &DatacentersRepr pattern well.
• Backfill workflow idempotency via chunk boundary hashing is correctly structured.

---

The GlobalDataKey::unpack missing DATA segment and the ensure_normal_if_missing unconditional upsert are the two issues that should be fixed before this merges.

[MasterPtato]

Merge activity

• Apr 13, 12:21 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Apr 13, 12:34 AM UTC: Graphite rebased this pull request as part of a merge.
• Apr 13, 12:34 AM UTC: @MasterPtato merged this pull request with Graphite.