Skip to content

Commit 4b6b5c6

Browse files
authored
feat(engine): add a load-testing harness for iterate-loop under concurrent load (#5781)
Reuses the fake-driver injection seam iterate-loop.test.ts already exercises to run many simulated tenant attempts through runIterateLoop concurrently, without spawning a real subprocess or spending API budget, and reports throughput per concurrency level. Publishes a baseline results doc alongside it. Closes #5224 Co-authored-by: nghetienhiep <13849419+nghetienhiep@users.noreply.github.com>
1 parent 7d59907 commit 4b6b5c6

5 files changed

Lines changed: 329 additions & 0 deletions

File tree

package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
"miner:env-reference": "node packages/loopover-miner/scripts/generate-env-reference.mjs",
2121
"miner:env-reference:check": "node packages/loopover-miner/scripts/generate-env-reference.mjs --check",
2222
"benchmark:miner": "node packages/loopover-miner/scripts/benchmark.mjs",
23+
"loadtest:iterate-loop": "npm run build --workspace @loopover/engine && node packages/loopover-engine/scripts/load-test-iterate-loop.mjs",
2324
"command-reference": "node scripts/gen-command-reference.mjs",
2425
"command-reference:check": "node scripts/gen-command-reference.mjs --check",
2526
"selfhost:validate-observability": "node scripts/validate-observability-configs.mjs",
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# iterate-loop load test
2+
3+
A committed load-testing harness for `runIterateLoop` (`src/miner/iterate-loop.ts`), the create->score->
4+
self-review->decide orchestrator AMS runs once per attempt. It reuses the same `CodingAgentDriver`
5+
injection seam `iterate-loop.test.ts` already exercises — the driver never spawns a real subprocess or
6+
spends API budget — but adds a configurable artificial per-iteration delay, so the numbers below measure
7+
iterate-loop's own orchestration/scheduling overhead under concurrent, multi-tenant-like load rather than a
8+
network call's latency. See issue #4913 for the parallel Worker-endpoint load-testing precedent this
9+
mirrors; #5224 is the AMS-side counterpart this harness was built for.
10+
11+
## Running it
12+
13+
```sh
14+
npm run loadtest:iterate-loop
15+
# or, from a workspace checkout, after building the engine:
16+
npm --workspace @loopover/engine run build
17+
node packages/loopover-engine/scripts/load-test-iterate-loop.mjs
18+
```
19+
20+
This prints a short text report to stdout and exits `0`. It does not fail the build or a CI job on its own
21+
— it is a signal to read, not a hard gate (there is no fixed pass/fail threshold, since wall-clock timing
22+
on shared CI runners is too noisy to gate on reliably). Run it locally before/after a change to
23+
`iterate-loop.ts`, `iterate-policy.ts`, `attempt-metering.ts`, or `self-review-adapter.ts` to see whether
24+
the change moved the needle under concurrency.
25+
26+
## What it measures
27+
28+
Each concurrency level runs a batch of simulated tenant attempts (a distinct `repoFullName`/
29+
`contributorLogin` per attempt, mirroring how independent tenants share the same AMS infra) through
30+
`runIterateLoop`, `concurrency` attempts in flight at a time via `Promise.all`, until the configured
31+
attempt count for that level completes. Every attempt is scripted to hand off after exactly one iteration
32+
(a passing self-review verdict on the first try), so the wall-clock numbers isolate the loop's own
33+
per-attempt overhead — driver invocation, self-review, policy decision, attempt-log append — from any
34+
variation in how many iterations a real attempt would take.
35+
36+
- **Concurrency levels:** 1, 8, 32, 128 concurrent attempts.
37+
- **Attempts per level:** 32 (script default) / 64 (baseline capture below).
38+
- **Simulated driver latency:** 15ms per iteration — a stand-in for the wall-clock a real coding-agent
39+
subprocess invocation would take, without actually spending any real API budget or spawning a process.
40+
41+
## Baseline (informational only, machine-dependent)
42+
43+
Captured on a Linux x86_64 dev container, Node.js 22.23.1, 64 attempts per level. Absolute numbers vary by
44+
hardware and by real driver/self-review latency — use this as a rough sense of scale and of how throughput
45+
scales with concurrency, not a target:
46+
47+
```
48+
iterate-loop load test
49+
50+
concurrency=1: 1023.87ms wall for 64 attempts, 63 attempts/sec, 64/64 handed off (simulated driver latency 15ms)
51+
concurrency=8: 151.74ms wall for 64 attempts, 422 attempts/sec, 64/64 handed off (simulated driver latency 15ms)
52+
concurrency=32: 65.20ms wall for 64 attempts, 982 attempts/sec, 64/64 handed off (simulated driver latency 15ms)
53+
concurrency=128: 40.59ms wall for 64 attempts, 1577 attempts/sec, 64/64 handed off (simulated driver latency 15ms)
54+
```
55+
56+
Throughput scales roughly linearly with concurrency up to the point where the batch size matches (or
57+
exceeds) the attempt count per level — at that point every attempt starts in the same tick and the
58+
per-attempt overhead is fully parallelized, bounded only by the simulated driver latency plus the loop's own
59+
synchronous work per attempt. This is execution/measurement only against the existing, already-injectable
60+
driver seam; it does not change `runIterateLoop`'s own concurrency model. These numbers feed the per-tenant
61+
scheduling and queue-fairness design work in the AMS Cloud Readiness milestone — reference them there rather
62+
than re-measuring.
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import type { CodingAgentDriver, CodingAgentDriverResult, CodingAgentDriverTask } from "../src/miner/coding-agent-driver.js";
2+
3+
export type LoadTestOptions = {
4+
levels?: number[];
5+
attemptCount?: number;
6+
latencyMs?: number;
7+
};
8+
9+
export type LoadTestLevelResult = {
10+
concurrency: number;
11+
attemptCount: number;
12+
latencyMs: number;
13+
wallMs: number;
14+
handoffCount: number;
15+
attemptsPerSecond: number;
16+
};
17+
18+
export declare const DEFAULT_CONCURRENCY_LEVELS: number[];
19+
export declare const DEFAULT_ATTEMPTS_PER_LEVEL: number;
20+
export declare const DEFAULT_SIMULATED_DRIVER_LATENCY_MS: number;
21+
22+
export declare function buildFakeLoadTestDriver(latencyMs: number): CodingAgentDriver & {
23+
run(task: CodingAgentDriverTask): Promise<CodingAgentDriverResult>;
24+
};
25+
26+
export declare function runConcurrencyLevel(
27+
concurrency: number,
28+
options?: LoadTestOptions,
29+
): Promise<LoadTestLevelResult>;
30+
31+
export declare function runLoadTest(options?: LoadTestOptions): Promise<LoadTestLevelResult[]>;
32+
33+
export declare function formatLoadTestReport(results: readonly LoadTestLevelResult[]): string;
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
#!/usr/bin/env node
2+
import { performance } from "node:perf_hooks";
3+
import { parseFocusManifest, runIterateLoop } from "../dist/index.js";
4+
5+
// Load-testing harness for the AMS iterate-loop orchestrator (#5224): every existing iterate-loop test is
6+
// correctness-oriented (single attempt, fake driver returns instantly), so there is no signal today for how
7+
// runIterateLoop behaves when many tenants' attempts run concurrently against shared infra. This harness
8+
// reuses the SAME fake-driver injection seam iterate-loop.test.ts already exercises (a CodingAgentDriver whose
9+
// `run()` never spawns a real subprocess or spends API budget) but adds a configurable artificial per-iteration
10+
// delay, so the measured throughput reflects iterate-loop's own orchestration/scheduling overhead under
11+
// concurrency rather than a network call's latency. See docs/iterate-loop-load-test.md (#5224) for how to run
12+
// this and read the numbers, and issue #4913 for the parallel Worker-endpoint load-testing precedent.
13+
14+
export const DEFAULT_CONCURRENCY_LEVELS = [1, 8, 32, 128];
15+
export const DEFAULT_ATTEMPTS_PER_LEVEL = 32;
16+
export const DEFAULT_SIMULATED_DRIVER_LATENCY_MS = 15;
17+
18+
const SYNTHETIC_ISSUE_NUMBER = 7;
19+
20+
/** One open issue per synthetic tenant repo, matching that tenant's own `passesPredictedGate` linkage below --
21+
* each tenant is a fully independent repo/contributor pair (`buildSelfReviewPredictedGateInput`'s own identity
22+
* fields), the same "multi-tenant-like" shape the issue's Problem section asks this harness to load-test. */
23+
function buildReviewContext(tenantRepoFullName) {
24+
return {
25+
manifest: parseFocusManifest({ gate: { duplicates: "block", linkedIssue: "advisory" } }),
26+
repo: { fullName: tenantRepoFullName, owner: tenantRepoFullName.split("/")[0], name: tenantRepoFullName.split("/")[1], isInstalled: true, isRegistered: true, isPrivate: false },
27+
issues: [{ repoFullName: tenantRepoFullName, number: SYNTHETIC_ISSUE_NUMBER, title: "Synthetic load-test issue", state: "open", labels: [], linkedPrs: [] }],
28+
pullRequests: [],
29+
};
30+
}
31+
32+
/** A `CodingAgentDriver` (coding-agent-driver.ts) that never spawns a real subprocess or spends API budget --
33+
* it resolves after `latencyMs` (simulating the wall-clock an iteration of a real coding-agent invocation
34+
* would take) with a scripted, always-passing result. `latencyMs` uses a real `setTimeout`, not a busy-loop, so
35+
* concurrent attempts genuinely interleave on the event loop the way concurrent live attempts would. */
36+
export function buildFakeLoadTestDriver(latencyMs) {
37+
return {
38+
async run(task) {
39+
await new Promise((resolve) => setTimeout(resolve, latencyMs));
40+
return { ok: true, changedFiles: [`src/${task.attemptId}.ts`], summary: `synthetic load-test change for ${task.attemptId}`, turnsUsed: 1 };
41+
},
42+
};
43+
}
44+
45+
const NOOP_SLOP_ASSESSMENT = { slopRisk: 0, band: "clean", findings: [] };
46+
47+
/** One simulated tenant attempt: a distinct `repoFullName`/`contributorLogin` per `tenantIndex`, a linked open
48+
* issue that matches on the first iteration, so every attempt hands off in exactly one iteration -- isolating
49+
* the measurement to iterate-loop's own per-attempt orchestration overhead rather than varying iteration counts
50+
* across runs. */
51+
async function runOneAttempt(tenantIndex, driver) {
52+
const repoFullName = `load-test-tenant-${tenantIndex}/repo`;
53+
const attemptId = `attempt-${tenantIndex}`;
54+
const input = {
55+
attemptId,
56+
workingDirectory: `/tmp/${attemptId}`,
57+
acceptanceCriteriaPath: `/tmp/${attemptId}/acceptance-criteria.json`,
58+
instructions: "Synthetic load-test instructions",
59+
mode: "live",
60+
maxIterations: 3,
61+
maxTurnsPerIteration: 20,
62+
repoFullName,
63+
contributorLogin: `miner-${tenantIndex}`,
64+
title: "Synthetic load-test attempt",
65+
body: `Closes #${SYNTHETIC_ISSUE_NUMBER}`,
66+
linkedIssues: [SYNTHETIC_ISSUE_NUMBER],
67+
reviewContext: buildReviewContext(repoFullName),
68+
rejectionSignaled: false,
69+
};
70+
const deps = {
71+
driver,
72+
runSlopAssessment: () => NOOP_SLOP_ASSESSMENT,
73+
appendAttemptLogEvent: () => {},
74+
};
75+
const start = performance.now();
76+
const result = await runIterateLoop(input, deps);
77+
return { elapsedMs: performance.now() - start, result };
78+
}
79+
80+
/**
81+
* Run `attemptCount` simulated tenant attempts concurrently (`Promise.all`, all started in the same tick) against
82+
* one shared fake driver, and report the aggregate wall time plus derived throughput. Every attempt is expected
83+
* to hand off after its first iteration (see {@link runOneAttempt}) -- a non-`"handoff"` outcome or a driver
84+
* error would silently understate real concurrent load, so both are counted and surfaced rather than ignored.
85+
*/
86+
export async function runConcurrencyLevel(concurrency, options = {}) {
87+
const attemptCount = options.attemptCount ?? DEFAULT_ATTEMPTS_PER_LEVEL;
88+
const latencyMs = options.latencyMs ?? DEFAULT_SIMULATED_DRIVER_LATENCY_MS;
89+
const driver = buildFakeLoadTestDriver(latencyMs);
90+
91+
const start = performance.now();
92+
const outcomes = [];
93+
for (let batchStart = 0; batchStart < attemptCount; batchStart += concurrency) {
94+
const batchSize = Math.min(concurrency, attemptCount - batchStart);
95+
const batch = await Promise.all(
96+
Array.from({ length: batchSize }, (_unused, offset) => runOneAttempt(batchStart + offset, driver)),
97+
);
98+
outcomes.push(...batch);
99+
}
100+
const wallMs = performance.now() - start;
101+
const handoffCount = outcomes.filter((o) => o.result.outcome === "handoff").length;
102+
103+
return {
104+
concurrency,
105+
attemptCount,
106+
latencyMs,
107+
wallMs,
108+
handoffCount,
109+
attemptsPerSecond: attemptCount / (wallMs / 1000),
110+
};
111+
}
112+
113+
/** Run every concurrency level in `levels` in sequence (never overlapping each other), so one level's
114+
* scheduling contention never bleeds into the next level's measurement. */
115+
export async function runLoadTest(options = {}) {
116+
const levels = options.levels ?? DEFAULT_CONCURRENCY_LEVELS;
117+
const results = [];
118+
for (const concurrency of levels) {
119+
results.push(await runConcurrencyLevel(concurrency, options));
120+
}
121+
return results;
122+
}
123+
124+
/** Render load-test results as a stable, greppable text report (no locale-dependent number formatting). */
125+
export function formatLoadTestReport(results) {
126+
const lines = ["iterate-loop load test", ""];
127+
for (const r of results) {
128+
lines.push(
129+
`concurrency=${r.concurrency}: ${r.wallMs.toFixed(2)}ms wall for ${r.attemptCount} attempts, ` +
130+
`${Math.round(r.attemptsPerSecond)} attempts/sec, ${r.handoffCount}/${r.attemptCount} handed off ` +
131+
`(simulated driver latency ${r.latencyMs}ms)`,
132+
);
133+
}
134+
return lines.join("\n");
135+
}
136+
137+
async function main() {
138+
const results = await runLoadTest();
139+
console.log(formatLoadTestReport(results));
140+
}
141+
142+
if (import.meta.url === `file://${process.argv[1]}`) {
143+
main();
144+
}
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
import { performance } from "node:perf_hooks";
2+
import { spawnSync } from "node:child_process";
3+
import { describe, expect, it } from "vitest";
4+
import {
5+
DEFAULT_ATTEMPTS_PER_LEVEL,
6+
DEFAULT_CONCURRENCY_LEVELS,
7+
DEFAULT_SIMULATED_DRIVER_LATENCY_MS,
8+
buildFakeLoadTestDriver,
9+
formatLoadTestReport,
10+
runConcurrencyLevel,
11+
runLoadTest,
12+
} from "../../packages/loopover-engine/scripts/load-test-iterate-loop.mjs";
13+
14+
describe("iterate-loop load-test script (#5224)", () => {
15+
it("the fake driver never spawns a real subprocess and resolves a scripted ok result after the configured latency", async () => {
16+
const driver = buildFakeLoadTestDriver(5);
17+
const start = performance.now();
18+
const result = await driver.run({
19+
attemptId: "attempt-0",
20+
workingDirectory: "/tmp/attempt-0",
21+
acceptanceCriteriaPath: "/tmp/attempt-0/acceptance-criteria.json",
22+
instructions: "synthetic",
23+
maxTurns: 1,
24+
});
25+
expect(performance.now() - start).toBeGreaterThanOrEqual(4);
26+
expect(result.ok).toBe(true);
27+
expect(result.changedFiles).toEqual(["src/attempt-0.ts"]);
28+
expect(result.turnsUsed).toBe(1);
29+
});
30+
31+
it("runs a small concurrency level end-to-end and every attempt hands off on its first iteration", async () => {
32+
const level = await runConcurrencyLevel(4, { attemptCount: 8, latencyMs: 1 });
33+
expect(level.concurrency).toBe(4);
34+
expect(level.attemptCount).toBe(8);
35+
expect(level.latencyMs).toBe(1);
36+
expect(level.handoffCount).toBe(8);
37+
expect(Number.isFinite(level.wallMs)).toBe(true);
38+
expect(level.wallMs).toBeGreaterThan(0);
39+
expect(Number.isFinite(level.attemptsPerSecond)).toBe(true);
40+
expect(level.attemptsPerSecond).toBeGreaterThan(0);
41+
});
42+
43+
it("runs a concurrency level where the batch size exceeds the attempt count in a single batch", async () => {
44+
const level = await runConcurrencyLevel(128, { attemptCount: 3, latencyMs: 1 });
45+
expect(level.attemptCount).toBe(3);
46+
expect(level.handoffCount).toBe(3);
47+
});
48+
49+
it("runs every concurrency level supplied via options.levels, in order", async () => {
50+
const results = await runLoadTest({ levels: [1, 2], attemptCount: 2, latencyMs: 1 });
51+
expect(results).toHaveLength(2);
52+
expect(results.map((r) => r.concurrency)).toEqual([1, 2]);
53+
for (const r of results) expect(r.handoffCount).toBe(2);
54+
});
55+
56+
it("exposes the documented default concurrency levels, attempt count, and simulated latency", () => {
57+
expect(DEFAULT_CONCURRENCY_LEVELS).toEqual([1, 8, 32, 128]);
58+
expect(DEFAULT_ATTEMPTS_PER_LEVEL).toBe(32);
59+
expect(DEFAULT_SIMULATED_DRIVER_LATENCY_MS).toBe(15);
60+
});
61+
62+
it("renders a deterministic report with no locale-dependent number formatting", () => {
63+
expect(
64+
formatLoadTestReport([
65+
{ concurrency: 1, attemptCount: 10, latencyMs: 15, wallMs: 160.4, handoffCount: 10, attemptsPerSecond: 62.34 },
66+
{ concurrency: 8, attemptCount: 10, latencyMs: 15, wallMs: 20.1, handoffCount: 9, attemptsPerSecond: 497.5 },
67+
]),
68+
).toBe(
69+
[
70+
"iterate-loop load test",
71+
"",
72+
"concurrency=1: 160.40ms wall for 10 attempts, 62 attempts/sec, 10/10 handed off (simulated driver latency 15ms)",
73+
"concurrency=8: 20.10ms wall for 10 attempts, 498 attempts/sec, 9/10 handed off (simulated driver latency 15ms)",
74+
].join("\n"),
75+
);
76+
});
77+
78+
it("runs end-to-end as a CLI script and prints the report header plus every default concurrency level", () => {
79+
const result = spawnSync(process.execPath, ["packages/loopover-engine/scripts/load-test-iterate-loop.mjs"], {
80+
cwd: process.cwd(),
81+
encoding: "utf8",
82+
});
83+
expect(result.status).toBe(0);
84+
expect(result.stdout).toContain("iterate-loop load test");
85+
for (const level of DEFAULT_CONCURRENCY_LEVELS) {
86+
expect(result.stdout).toContain(`concurrency=${level}:`);
87+
}
88+
});
89+
});

0 commit comments

Comments
 (0)