Skip to content

se6-module-loading-perf-audit.md

Latest commit

 

History

History
57 lines (49 loc) · 13.2 KB

se6-module-loading-perf-audit.md

File metadata and controls

57 lines (49 loc) · 13.2 KB

se6 module-loading-perf Audit For Agent OS

Scope:

  • Compared /home/nathan/se6 main...module-loading-perf.
  • Covered every substantive change set from f32fdfd3 through 3a46a184.
  • Collapsed the many generated benchmark output files under packages/secure-exec/benchmarks/results/module-load/** into a single artifact row instead of listing every generated JSON/Markdown file separately.
  • Line references below are repo-relative and point at the branch tip for se6 and the current Agent OS checkout in a6.

Status legend:

  • Applies: still missing in Agent OS and worth porting.
  • Partially applies: the idea still matters, but the original implementation targeted old secure-exec/V8 bridge architecture.
  • Already absorbed: Agent OS already has a native equivalent.
  • Compat-only: only relevant to the legacy compat runtime under packages/core/src/compat-runtime/**.
  • No longer applies: tied to old secure-exec transport/bootstrap mechanics that no longer exist in native Agent OS.
  • Docs-only: no runtime delta to port.

Change Matrix

Change set Status in Agent OS Secure-exec refs Agent OS refs Migration note
f32fdfd3 Initial IPC observability + module-load benchmark Partially applies se6/packages/secure-exec/benchmarks/module-load/run-module-load-benchmarks.ts:52-58,96-104,458-525
se6/packages/secure-exec/benchmarks/module-load/summary.ts:420-520,1155-1183		 a6/crates/execution/src/benchmark.rs:73-179,333-470
a6/crates/execution/src/bin/node-import-bench.rs:1-40
a6/crates/execution/src/node_import_cache.rs:53-63,65-176		 Agent OS already has a native node-import benchmark and import-cache metrics, but it does not have the old ndjson/prometheus/summary/comparison artifact stack.
41215f48 US-001 real Pi CLI end-to-end benchmark workload Partially applies se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:149-162 a6/scripts/benchmarks/bench-utils.ts:64-192
a6/scripts/benchmarks/coldstart.bench.ts:4-16,81-114
a6/packages/core/src/agent-os.ts:2359-2416		 Agent OS benchmarks createSession("pi"), but not the old standalone NodeRuntime Pi CLI path that secure-exec was timing.
3772f039 US-002 machine-readable summaries and deltas Applies se6/packages/secure-exec/benchmarks/module-load/run-module-load-benchmarks.ts:505-523
se6/packages/secure-exec/benchmarks/module-load/summary.ts:2688-2823		 a6/crates/execution/src/bin/node-import-bench.rs:3-19
a6/crates/execution/src/benchmark.rs:73-179		 Current Agent OS benchmark prints Markdown only. There is no native JSON summary or before/after comparison artifact yet.
9adad215 US-003 raw UDS RTT + per-session phase attribution Applies se6/packages/secure-exec/benchmarks/module-load/run-module-load-benchmarks.ts:52-58
se6/packages/secure-exec/benchmarks/module-load/summary.ts:437-455,1695-1705,2328-2338,2776-2790		 a6/crates/execution/src/benchmark.rs:50-61,181-245
a6/crates/sidecar/src/protocol.rs:737-817,1054-1095		 Agent OS can validate typed request/response flow, but it does not benchmark transport RTT or break time into create/inject/execute/destroy phases.
6197d51a US-011 direct Pi CLI headless execution inside standalone NodeRuntime Partially applies se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:149-162 a6/packages/core/src/agent-os.ts:2359-2416
a6/packages/core/tests/pi-headless.test.ts:129-138		 createSession("pi") exists natively, but the old standalone-runtime headless path is not the main Agent OS target. The legacy V8/compat PI headless TODO is still open.
88d1992e US-012 JSZip + pdf-lib scenarios Applies se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:101-130 a6/crates/execution/src/benchmark.rs:422-469
a6/crates/execution/tests/benchmark.rs:3-59		 Native Agent OS benchmark still covers only five synthetic scenarios.
4bb099df US-013 repeated-session compressed JSZip benchmarking Applies se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:118-130
se6/packages/secure-exec/benchmarks/module-load/orchestration.ts:17-24,203-253		 a6/crates/execution/src/benchmark.rs:422-469 This only makes sense if the richer module-load suite is ported into Agent OS.
4aae2210 US-004 eliminate repeated _loadPolyfill round-trips Compat-only se6/packages/core/isolate-runtime/src/inject/require-setup.ts:4024-4041,4048-4090,4573-4589 a6/packages/core/src/compat-runtime/kernel/isolate-runtime/require-setup.js:4008-4024
a6/crates/execution/src/node_import_cache.rs:65-176		 The native Agent OS Node path no longer uses _loadPolyfill; only the legacy compat runtime still does.
15fe7482 US-014 split _loadPolyfill attribution from __bd:* dispatch Compat-only se6/packages/secure-exec/benchmarks/module-load/summary.ts:1129-1146,1364-1445,2696-2823 a6/packages/core/src/compat-runtime/nodejs/bridge.js:7027-7072 Useful only if compat-runtime observability is ported. It does not map to the native Node executor.
96164a4c US-005 shrink _loadPolyfill payload transfer with id/hash caching Compat-only se6/packages/v8/src/runtime.ts:110-230,729-761,793-805 a6/packages/core/src/compat-runtime/kernel/isolate-runtime/require-setup.js:4008-4024
a6/packages/core/src/compat-runtime/nodejs/bridge-setup.js:1718-1729		 Native Agent OS uses filesystem-backed loader assets and env wiring, not repeated polyfill payload IPC.
91c688ce US-015 restore full-suite jszip-end-to-end completion Applies se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:125-130
se6/packages/secure-exec/benchmarks/module-load/orchestration.ts:17-24,431-470		 a6/crates/execution/src/benchmark.rs:422-469 The scenario is simply absent from the native benchmark suite today.
a5f06534 US-006 stop resending large bridge bootstrap payloads No longer applies se6/packages/v8/src/runtime.ts:131-137,620-663,793-805,816-833
se6/native/v8-runtime/src/snapshot.rs:143-220		 a6/crates/execution/src/javascript.rs:364-380,403-550
a6/crates/execution/src/node_import_cache.rs:2410-2450		 Agent OS materializes loader/bootstrap files once and passes paths/env to Node. It is not shipping bridge source blobs per execution.
9efe5dc1 US-016 dedupe static postRestoreScript bytes No longer applies se6/packages/v8/src/runtime.ts:135-137,614-663,799-833 a6/crates/execution/src/javascript.rs:528-550 There is no native Agent OS equivalent to secure-exec's postRestoreScript payload channel.
be83b6fd US-007 cache module resolution, package metadata, and filesystem probes Already absorbed se6/packages/core/isolate-runtime/src/inject/require-setup.ts:4034-4090 a6/crates/execution/src/node_import_cache.rs:53-63,65-176,487-603,690-775 This is already present natively as NodeImportCache with persisted resolution/module-format/package-type caches and validation.
834a057f US-008 preload or snapshot hottest bootstrap assets Mostly absorbed se6/native/v8-runtime/src/snapshot.rs:15-26,143-220
se6/packages/core/isolate-runtime/src/inject/require-setup.ts:4029-4033,4578-4589		 a6/crates/execution/src/javascript.rs:17-42,329-380,553-619
a6/crates/execution/src/node_import_cache.rs:2332-2450		 Agent OS already materializes builtin/polyfill assets and prewarms hot imports. The part that remains unported is the broader warm-pool/snapshot layer.
d22ee524 US-018 bypass _loadPolyfill for hot __bd:* bridge-dispatch wrappers Compat-only se6/packages/nodejs/src/execution-driver.ts:713-776 a6/packages/core/src/compat-runtime/nodejs/bridge.js:7027-7072 Native Agent OS does not route bridge calls through __bd:*. This is only relevant if the compat runtime stays performance-critical.
7f2467c1 US-019 preload or manifest-cache hot projected package source files Partially applies se6/packages/core/isolate-runtime/src/inject/require-setup.ts:4034-4090
se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:79-85		 a6/packages/core/tests/software-projection.test.ts:34-72
a6/crates/execution/src/node_import_cache.rs:179-196,487-530,690-775		 Projected packages exist and metadata caching exists, but there is no explicit manifest/preload layer for the hottest projected source files.
ba7f25d5 US-019 docs-only follow-up Docs-only se6/CLAUDE.md
se6/scripts/ralph/prd.json
se6/scripts/ralph/progress.txt		 a6/crates/execution/benchmarks/node-import-baseline.md:1-24 No runtime delta. This was only benchmark planning/progress tracking.
c12515b2 US-020 explicit session-destroy acknowledgment Already absorbed se6/packages/v8/src/runtime.ts:380-389,893-940
se6/native/v8-runtime/src/main.rs:227-255		 a6/crates/sidecar/src/service.rs:1066-1071,1933-2015
a6/crates/sidecar/src/protocol.rs:737-817,1054-1095		 Agent OS already has typed request/response expectations and a direct disposal path. There is no ping/pong teardown pattern to port.
1f83e050 US-009 reduce fixed per-session overhead Docs-only se6/CLAUDE.md
se6/docs-internal/todo.md
se6/scripts/ralph/prd.json
se6/scripts/ralph/progress.txt		 a6/crates/execution/src/benchmark.rs:209-243
a6/crates/execution/benchmarks/node-import-baseline.md:21-24		 The secure-exec commit was planning only. The goal still matters in Agent OS, but there is no direct code patch to transplant.
11a33bb0 US-010 expand benchmark controls for cold, warm, and host comparisons Applies se6/packages/secure-exec/benchmarks/module-load/orchestration.ts:17-24,38-50,203-253,431-470
se6/packages/secure-exec/benchmarks/module-load/summary.ts:2748-2790		 a6/crates/execution/src/benchmark.rs:422-469
a6/crates/execution/tests/benchmark.rs:16-24		 Native Agent OS has no mode matrix for true cold start, same-session replay, snapshot on/off, or host controls.
71fde781 US-021 ranked bridge-target hotspots in summaries Applies se6/packages/secure-exec/benchmarks/module-load/summary.ts:1364-1445,2696-2705,2823-2832 a6/crates/execution/src/benchmark.rs:141-245 Agent OS currently emits guidance prose, not ranked hotspot tables or target-level deltas.
a8a9fabc US-024 stabilize top-level module-load benchmark orchestration Applies se6/packages/secure-exec/benchmarks/module-load/orchestration.ts:52-111,431-470 a6/crates/execution/src/benchmark.rs:333-350 The current native harness is simple and single-stage. The old staged orchestration only matters if the broader suite is ported.
0f9e3096 US-022 isolation microbenchmarks for empty session and hot single imports Partially applies se6/packages/secure-exec/benchmarks/module-load/scenario-catalog.ts:21-85 a6/crates/execution/src/benchmark.rs:422-469
a6/crates/execution/tests/benchmark.rs:16-24		 Agent OS already has isolate-startup, builtin-import, local-graph, and large-package cases, but not the fuller hot-import microbench set (stream, stream/web, crypto, zlib, assert, url, projected package files).
3a46a184 US-023 stability + resource-usage reporting Partially applies se6/packages/secure-exec/benchmarks/module-load/summary.ts:432-455,1690-1705,2331-2342,2721-2745
se6/packages/secure-exec/benchmarks/module-load/run-module-load-benchmarks.ts:467-480		 a6/scripts/benchmarks/memory.bench.ts:45-56,117-175,206-241
a6/crates/execution/src/benchmark.rs:73-179		 Agent OS has a separate memory benchmark, but the native node-import benchmark does not integrate per-scenario stability or host resource usage into one report.
Generated benchmark results under packages/secure-exec/benchmarks/results/module-load/** Do not port wholesale se6/packages/secure-exec/benchmarks/results/module-load/** a6/crates/execution/benchmarks/node-import-baseline.md:1-24 Keep only representative baselines in Agent OS. The generated result tree is reference data, not code to integrate.
Packaging/docs/test plumbing around the old benchmark runner Mostly no longer applies se6/packages/secure-exec/tests/module-load-summary.test.ts
se6/CLAUDE.md
se6/scripts/ralph/prd.json
se6/scripts/ralph/progress.txt		 a6/crates/execution/src/bin/node-import-bench.rs:1-40
a6/scripts/benchmarks/coldstart.bench.ts:1-116		 Only port the parts that support a deliberate native benchmark-suite expansion. Most secure-exec package plumbing was specific to the old repo layout and reporting workflow.

What I Would Actually Port

  1. Port the benchmark/reporting pieces first: US-002, US-003, US-010, US-021, US-022, US-023, and the orchestration parts of US-024.
  2. Then add the missing real-library workloads from US-012, US-013, and US-015.
  3. Treat US-007, US-008, and US-020 as already largely or fully absorbed by the native Agent OS runtime.
  4. Do not port US-004, US-005, US-014, or US-018 into the native path unless the legacy compat runtime under packages/core/src/compat-runtime/** becomes a deliberate optimization target.
  5. Do not port generated benchmark output trees wholesale. Keep a small native baseline artifact instead.