Skip to content

Commit 2bbe577

Browse files
committed
feat(gateways): one deployment shape, and a pre-deploy check that it holds
Every gateway is now deployed the same way: drop its config files, set its env, set its headers, run its commands, and the harness states the core limits last. Nothing else, and the engine understands none of it. COMMANDS, for the one gateway that has no config file. It keeps its configuration in a database and is configured through its own admin API after boot, so there is nothing to drop next to the binary and no honest way to deploy it except to make the calls its documentation describes. `commands` is a file of lines, discovered by name exactly like `env` and `headers.json`, run in order once the gateway answers as ready and before anything is measured. The engine does not read what they write and does not know what they do, the same way it does not know what a header does. definition.json is unchanged. THE HARNESS STATES THE CORE COUNT, not the gateway. A cpuset restricts which cores a process may use; it does not change what a runtime believes is available, and a runtime that reads the machine's CPU count sizes its pools for sixteen cores while confined to four. That was left to each gateway to remember and it drifted exactly as that always does: both manifests that set it lost the setting in the port and nothing noticed, because the gateway still booted. The harness now appends the limits after the gateway's own env, so a gateway cannot opt out of the size it is being measured at. kong and gomodel get their env back. Both launched with empty env blocks for the same reason, which is why one never bound its port and the other started with no upstream at all. apisix's route template contained a shell function call that the retired manifest used to expand. Nothing expands it now, so the literal text went into the YAML, the routes would not parse, and the gateway answered every probe 404: it published as an entrant that serves nothing. Expanded to data, and validate now refuses a config template containing a command substitution, because a config is data and nothing runs to produce one. THE NECESSITY RULE IS CHECKED AGAIN. The published methodology promises a setting appears only if it boots the process, points an upstream at the mock, exposes an ingress path, or binds the port and cores. The lint enforcing that read its claims from GW_CONFIG_WHY inside gateway.sh, which is dead code nothing loads, so the claims described a deployment that no longer existed and the parts that grew afterwards were covered by nothing. All 294 claims move into a `why` file per gateway, and lib/gateway_deploy_lint.sh checks env, headers and commands against them. Proven by planting an unjustified setting and watching it fail.
1 parent 1dd33a1 commit 2bbe577

23 files changed

Lines changed: 645 additions & 6 deletions

File tree

‎engine/src/bin/otb.rs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,30 @@ fn main() -> ExitCode {
321321
match otb_engine::launch::launch_default(&mut launcher, &spec) {
322322
Ok(l) => {
323323
println!("launched {} in {} attempt(s)", spec.runtime.identity(), l.attempts);
324+
// COMMANDS, in order, now that it is up and before anything is measured.
325+
//
326+
// A gateway with no config file is configured through its own admin API
327+
// after it boots, so this is the only point at which that can happen: it
328+
// needs the process answering, and it must be finished before a probe
329+
// decides what the gateway does or does not serve.
330+
//
331+
// A failure here stops the run. A half-configured gateway is worse than
332+
// one that never started: it answers probes, and publishes a verdict for
333+
// an upstream that was never wired up.
334+
for line in &cfg.manifest.commands {
335+
match otb_engine::launch::run_line(line, Duration::from_secs(120)) {
336+
Ok(()) => println!("setup: {line}"),
337+
Err(why) => {
338+
eprintln!("setup command failed: {line}: {why}");
339+
let _ = otb_engine::supervise::stop_and_wait(
340+
&spec.runtime,
341+
spec.port,
342+
Duration::from_secs(15),
343+
);
344+
return ExitCode::FAILURE;
345+
}
346+
}
347+
}
324348
Some(spec)
325349
}
326350
Err(e) => {

‎engine/src/config_lint.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,7 @@ mod tests {
175175
headers: vec![],
176176
runtime: Runtime::Docker { container: "gw-bench".into() },
177177
egress: vec!["openai".into()],
178+
commands: vec![],
178179
config: vec![],
179180
launch: None,
180181
config_files: vec![],

‎engine/src/launch.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -351,6 +351,15 @@ pub trait Launcher {
351351
}
352352
}
353353

354+
/// Run ONE line from a gateway's `commands` file, through a shell, with a hard timeout.
355+
///
356+
/// A shell because these lines are what a gateway's own documentation tells an operator to type,
357+
/// and those instructions use pipes, quoting and redirection. The alternative is a bespoke argv
358+
/// encoding that every author would have to learn in order to transcribe a documented curl.
359+
pub fn run_line(line: &str, timeout: Duration) -> Result<(), String> {
360+
run_with_timeout("/bin/sh", &["-c".to_string(), line.to_string()], timeout)
361+
}
362+
354363
/// Run a pre-launch command with its own hard timeout, using only `std::process`: poll
355364
/// `Child::try_wait` rather than blocking on `wait`, so a hung command is killed instead of hanging
356365
/// this call forever.

‎engine/src/manifest.rs‎

Lines changed: 148 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,23 @@ pub struct Manifest {
9595
/// regardless and publishes what it observes. This only says which upstreams are wired.
9696
#[serde(default)]
9797
pub egress: Vec<String>,
98+
/// COMMANDS RUN AFTER THE GATEWAY IS UP, one per line, from a file named `commands`.
99+
///
100+
/// Discovered by filename like `env` and `headers.json`, never declared, so every gateway is
101+
/// described the same way and a reader can see the whole deployment by listing the directory.
102+
///
103+
/// This exists for one shape of gateway: the kind with no config file at all, which stores its
104+
/// configuration in a database and is configured through its own admin API after it boots. There
105+
/// is no file to drop for those, so the only honest way to deploy one the way its operators do
106+
/// is to make the same calls its documentation tells you to make. Almost every gateway has no
107+
/// `commands` file and needs none.
108+
///
109+
/// Lines are run in order, each through a shell, after the gateway answers as ready and before
110+
/// anything is measured. A line that fails fails the run: a gateway configured halfway is worse
111+
/// than one that never started, because it will answer probes and publish numbers for an
112+
/// upstream that was never wired up.
113+
#[serde(default, skip)]
114+
pub commands: Vec<String>,
98115
#[serde(default)]
99116
pub config: Vec<ConfigSetting>,
100117
/// Headers that select the EGRESS, keyed by dialect.
@@ -360,6 +377,42 @@ impl std::fmt::Display for ManifestError {
360377
}
361378
}
362379

380+
/// PARALLELISM, SET BY THE HARNESS FOR EVERY GATEWAY, FROM THE CORES IT PINNED.
381+
///
382+
/// The harness decides how many cores a gateway gets. It follows that the harness, not the gateway,
383+
/// must tell the gateway's runtime how many it got, and must tell every gateway the same way.
384+
///
385+
/// Pinning alone is not enough, and the gap is silent. A cpuset restricts which cores a process may
386+
/// run on; it does not change what a runtime THINKS is available. Some runtimes ask the kernel for
387+
/// their affinity and get the right answer; others read the machine's online CPU count and size
388+
/// their thread pools for a sixteen core box while confined to four. That gateway then runs with
389+
/// four times the threads it should have, contends with itself, and publishes a number that
390+
/// describes a configuration no operator would deploy.
391+
///
392+
/// This was previously left to each gateway to remember in its own env block, and it drifted exactly
393+
/// as that arrangement always does: the two manifests that set it lost the setting entirely when
394+
/// they were ported, and nothing noticed because the gateway still booted.
395+
///
396+
/// The list is deliberately runtime-standard names only. A gateway whose runtime has its own knob
397+
/// declares it in its own env with `{NCORE}`, because naming one entrant's variable in shared code
398+
/// would be per-gateway logic in the one place that must not have any.
399+
fn pinned_parallelism(ncore: u32) -> Vec<(String, String)> {
400+
let n = ncore.to_string();
401+
[
402+
// Go: honours affinity already, set anyway so the value is explicit and identical everywhere.
403+
"GOMAXPROCS",
404+
// Node/libuv worker pool.
405+
"UV_THREADPOOL_SIZE",
406+
// Rust rayon.
407+
"RAYON_NUM_THREADS",
408+
// OpenMP, which numeric python stacks size their pools from.
409+
"OMP_NUM_THREADS",
410+
]
411+
.iter()
412+
.map(|k| ((*k).to_string(), n.clone()))
413+
.collect()
414+
}
415+
363416
impl Manifest {
364417
pub fn validate(&self) -> Result<(), ManifestError> {
365418
for (v, field) in [
@@ -577,7 +630,10 @@ impl Manifest {
577630
};
578631
crate::launch::LaunchKind::Docker {
579632
image: image.clone(),
580-
env,
633+
// THE HARNESS WINS. The gateway's own env goes first and the pinning values
634+
// last, because a later assignment overrides an earlier one and no entrant may
635+
// opt out of the core limit it is being measured under.
636+
env: env.into_iter().chain(pinned_parallelism(core_count(cores))).collect(),
581637
// Every entrant uses host networking: the gateway binds the port it declares and
582638
// the harness drives that port. A published mapping would put a NAT hop inside
583639
// every measured request.
@@ -634,7 +690,8 @@ impl Manifest {
634690
crate::launch::LaunchKind::Native {
635691
binary: bin.to_string_lossy().into_owned(),
636692
args,
637-
env,
693+
// Same precedence as the container path: the harness states the core limit last.
694+
env: env.into_iter().chain(pinned_parallelism(core_count(cores))).collect(),
638695
env_unset: env_unset.clone(),
639696
}
640697
}
@@ -698,9 +755,26 @@ impl Manifest {
698755
));
699756
continue;
700757
}
758+
// A CONFIG TEMPLATE IS DATA, AND MUST NOT ASK FOR A COMMAND TO BE RUN.
759+
//
760+
// A gateway is a directory of data: static config files, an env block, headers. Nothing
761+
// executes to produce a config. Templates carried over from the retired shell manifests
762+
// still contained `$(...)`, which that shell expanded by calling a function; the engine
763+
// substitutes `{PLACEHOLDER}` and nothing else, so it wrote the text out verbatim and
764+
// the gateway got a config file with a shell fragment where a provider block belonged.
765+
//
766+
// It failed silently in the worst possible way. The gateway booted, bound its port,
767+
// answered every probe 404, and published as an entrant that serves nothing at all.
701768
// A placeholder the harness cannot supply. Caught here rather than at launch, where it
702769
// becomes a gateway booting with a literal `{MOCK_PORT}` in an upstream URL.
703770
if let Ok(raw) = std::fs::read_to_string(&t) {
771+
if let Some(bad) = raw.lines().find(|l| l.contains("$(")) {
772+
out.push(format!(
773+
"{}: contains a shell command substitution, which nothing will expand: {:?}. A config template is data; write the literal value or use a {{PLACEHOLDER}}",
774+
f.template,
775+
bad.trim()
776+
));
777+
}
704778
if let Err(e) = self.substitute(&raw, "0-3", 8000, gw_dir) {
705779
out.push(format!("{}: {e}", f.template));
706780
}
@@ -772,6 +846,22 @@ impl Manifest {
772846
m.apply_env(env, unset);
773847
}
774848

849+
// One command per line. Blank lines and `#` comments are skipped so the file can explain
850+
// itself, which matters more here than anywhere else: these are the only place the harness
851+
// does something to a gateway rather than handing it a file.
852+
let commands_path = dir.join("commands");
853+
if commands_path.is_file() {
854+
let raw = std::fs::read_to_string(&commands_path).map_err(|e| {
855+
ManifestLoadError::Unreadable { path: commands_path.clone(), why: e.to_string() }
856+
})?;
857+
m.commands = raw
858+
.lines()
859+
.map(str::trim)
860+
.filter(|l| !l.is_empty() && !l.starts_with('#'))
861+
.map(str::to_string)
862+
.collect();
863+
}
864+
775865
let headers_path = dir.join("headers.json");
776866
if headers_path.is_file() {
777867
let raw = std::fs::read_to_string(&headers_path)
@@ -863,6 +953,7 @@ impl Manifest {
863953
#[cfg(test)]
864954
mod tests {
865955
use super::*;
956+
use std::time::Duration;
866957

867958
fn docker_manifest() -> Manifest {
868959
Manifest {
@@ -878,6 +969,7 @@ mod tests {
878969
headers: vec![],
879970
runtime: Runtime::Docker { container: "gw-bench".into() },
880971
egress: vec!["openai".into()],
972+
commands: vec![],
881973
config: vec![],
882974
launch: None,
883975
config_files: vec![],
@@ -995,6 +1087,59 @@ mod tests {
9951087
fn the_url_is_built_from_the_declared_port_and_path() {
9961088
assert_eq!(docker_manifest().url(), "http://127.0.0.1:8080/v1/chat/completions");
9971089
}
1090+
1091+
// EVERY GATEWAY IS TOLD ITS CORE COUNT, BY THE HARNESS, THE SAME WAY.
1092+
//
1093+
// A cpuset restricts which cores a process may use; it does not change what a runtime believes
1094+
// is available. A runtime that reads the machine's online CPU count sizes its thread pools for a
1095+
// sixteen core box while confined to four, contends with itself, and publishes a number that
1096+
// describes a configuration nobody would deploy.
1097+
//
1098+
// This used to be each gateway's job to remember in its own env, and it drifted exactly as that
1099+
// always does: both manifests that set it lost the setting when they were ported and nothing
1100+
// noticed, because the gateway still booted. It is the harness that decides the core count, so
1101+
// it is the harness that states it, identically, everywhere.
1102+
#[test]
1103+
fn every_launched_gateway_is_told_how_many_cores_it_was_pinned_to() {
1104+
// Both launch kinds, because a native entrant is pinned with taskset rather than a cpuset
1105+
// and needs telling just as much.
1106+
let mut container = docker_manifest();
1107+
container.launch = Some(LaunchDecl::Docker {
1108+
image: "gw:1".into(),
1109+
env: vec![("GOMAXPROCS".into(), "64".into())],
1110+
args: vec![],
1111+
mounts: vec![],
1112+
});
1113+
let mut native = docker_manifest();
1114+
native.runtime = Runtime::Native { proc_match: "gw".into() };
1115+
native.launch = Some(LaunchDecl::Native {
1116+
build: None,
1117+
binary: vec!["gw".into()],
1118+
args: vec![],
1119+
env: vec![],
1120+
env_unset: vec![],
1121+
});
1122+
for (label, m) in [("container", container), ("native", native)] {
1123+
let dir = std::path::Path::new(".");
1124+
let spec = m
1125+
.launch_spec("0-3", 8000, dir, Duration::from_secs(1), Duration::from_secs(1))
1126+
.expect("this manifest declares a launch")
1127+
.expect("and it resolves");
1128+
let env = match &spec.kind {
1129+
crate::launch::LaunchKind::Docker { env, .. } => env,
1130+
crate::launch::LaunchKind::Native { env, .. } => env,
1131+
};
1132+
let got = |k: &str| env.iter().rev().find(|(n, _)| n == k).map(|(_, v)| v.as_str());
1133+
// "0-3" is four cores, and that is what the runtime must be told, not the host's count.
1134+
// Read the LAST assignment for each name: the harness appends its values after the
1135+
// gateway's, so a gateway that sets its own GOMAXPROCS cannot escape the pinning.
1136+
assert_eq!(got("GOMAXPROCS"), Some("4"), "{label}: GOMAXPROCS");
1137+
assert_eq!(got("UV_THREADPOOL_SIZE"), Some("4"), "{label}: node pool");
1138+
assert_eq!(got("RAYON_NUM_THREADS"), Some("4"), "{label}: rayon");
1139+
assert_eq!(got("OMP_NUM_THREADS"), Some("4"), "{label}: openmp");
1140+
}
1141+
}
1142+
9981143
}
9991144

10001145
#[cfg(test)]
@@ -1320,4 +1465,5 @@ mod real_field_tests {
13201465
assert!(u.contains(&m.port.to_string()));
13211466
}
13221467
}
1468+
13231469
}

‎engine/src/suite.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -691,6 +691,7 @@ mod tests {
691691
headers: vec![],
692692
runtime: crate::manifest::Runtime::Docker { container: "gw-bench".into() },
693693
egress: vec![],
694+
commands: vec![],
694695
config: vec![],
695696
launch: None,
696697
config_files: vec![],

‎gateways/agentgateway/why‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Why every setting, env var, header and command this gateway uses is here.
2+
#
3+
# The rule, from the published methodology: a setting may appear only if it is needed to boot the
4+
# process at all, to point an upstream at the test mock instead of a real provider, to expose an
5+
# ingress path the matrix exercises, or to bind the port and cores the rig requires. One claim per
6+
# line: <pattern> <boot|upstream|ingress|bind>. Anything unclaimed is a lint failure, and a claim
7+
# that matches nothing is a stale claim and also a failure.
8+
#
9+
# Migrated from GW_CONFIG_WHY in gateway.sh, which is dead code the lint was still reading.
10+
llm boot # the top-level block; without it no LLM surface is mounted at all
11+
port bind # the port the harness drives
12+
models boot # ModelRouter needs at least one entry to resolve a request model
13+
name upstream # the request model name that selects this upstream
14+
provider upstream # which provider dialect this entry egresses in
15+
params boot # required wrapper for the per-provider settings below
16+
model upstream # the model id sent to the mock in that provider's own shape
17+
baseUrl upstream # -> the mock. MUST carry a path (see _agentgw_write_config): a bare origin
18+
apiKey boot # a provider entry with no key is not registered (dummy; the mock ignores it)
19+
awsRegion boot # local.rs: \"bedrock requires aws_region\" - boot fails without it
20+
AWS_ACCESS_KEY_ID boot # the bedrock provider signs SigV4 from the AWS env; absent = no egress
21+
AWS_SECRET_ACCESS_KEY boot
22+
AWS_REGION boot

‎gateways/aisix/why‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Why every setting, env var, header and command this gateway uses is here.
2+
#
3+
# The rule, from the published methodology: a setting may appear only if it is needed to boot the
4+
# process at all, to point an upstream at the test mock instead of a real provider, to expose an
5+
# ingress path the matrix exercises, or to bind the port and cores the rig requires. One claim per
6+
# line: <pattern> <boot|upstream|ingress|bind>. Anything unclaimed is a lint failure, and a claim
7+
# that matches nothing is a stale claim and also a failure.
8+
#
9+
# Migrated from GW_CONFIG_WHY in gateway.sh, which is dead code the lint was still reading.
10+
resources_file boot # standalone source; without it aisix demands an etcd cluster
11+
proxy bind
12+
addr bind # the port the harness drives
13+
admin boot # with no admin block the binary refuses to load its config:
14+
admin_keys boot # \"admin.admin_keys must contain at least one key\" (tested, see below)
15+
_format_version boot # the resources file will not load without it
16+
provider_keys boot
17+
display_name boot # a resource without a name cannot be referenced
18+
provider upstream # which upstream dialect this key speaks
19+
adapter upstream # and which adapter builds its URL
20+
api_base upstream # -> the mock
21+
api_key boot # dummy; the mock ignores it
22+
models ingress # the client-facing model table
23+
model_name upstream # the model id sent upstream
24+
provider_key upstream # which provider_key (and so which egress) this model resolves to
25+
api_keys boot # aisix has NO anonymous mode: every proxy handler 401s without a known key
26+
key_env boot # where the plaintext comes from
27+
allowed_models boot # an api_key with no allowance can reach nothing
28+
BENCH_AISIX_KEY boot # that plaintext. The name must NOT start with AISIX_ - see _aisix_launch
29+
# The build-time variables are UNSET before launch (a leading `-` in `env` removes one), so the
30+
# measured process does not inherit the repo and revision its build used. They are claimed here
31+
# because the env file names them, and anything the deployment names must be justified.
32+
AISIX_REPO boot # unset at launch: build input, must not leak into the measured process
33+
AISIX_COMMIT boot # unset at launch: same
34+
AISIX_SRC boot # unset at launch: same

0 commit comments

Comments
 (0)