Project lightning: compiled graphs generated from procedurally wired ones (#726)

[0-jake-0]

The whole of project lightning in one PR. Supersedes #770 and the earlier ten-deep stack (#759–#768), all now closed. 25 files, +5014/−11.

What this changes

A graph whose shape comes from a config file can now be compiled.

#[wiring]
fn desk(g: &GraphBuilder, book: &[Instrument]) -> Stream<f64> {
    book.iter()
        .map(|inst| {
            let fee = inst.fee;
            g.ticker(inst.period).count().map(move |n: &u64| *n as f64 - fee)
        })
        .reduce(|a, b| a.join(&b, |x: &f64, y: &f64| x + y))
        .expect("at least one instrument")
}

Pass 1 runs that against the interpreted builder and walks the graph it built; pass 2 is an ordinary cargo build over the result:

wingfoil::nitro! {
    fn desk_generated(g: &GraphBuilder) -> Stream<f64> {
        let n0_ticker = g.ticker(::core::time::Duration::new(0u64, 1000000u32));
        let n1_count = n0_ticker.count();
        let n2_map = n1_count.map({ let fee = 0.25f64; move |n: &u64| *n as f64 - fee });
        let n3_ticker = g.ticker(::core::time::Duration::new(0u64, 4000000u32));
        let n4_count = n3_ticker.count();
        let n5_map = n4_count.map({ let fee = 1.75f64; move |n: &u64| *n as f64 - fee });
        let n6_join = n2_map.join(&n5_map, |x: &f64, y: &f64| x + y);
        n6_join
    }
}

The loop is gone — unrolled into one pipeline per instrument, each carrying the parameters it was configured with. nitro! never sees a loop.

Why

Implements #726. nitro! reads tokens, so its topology is fixed at compile time — the straight-line rule dual_mode states. This buys the one thing that rules out, topology decided by running code, while keeping nitro! as the sole backend: there remains exactly one place that knows how to turn wiring into a monomorphized runner, and the generator emits input to it rather than runner code, so the two cannot drift.

The engine erases closures (Box<dyn FnMut> over Rc slots), so a traversal recovers topology but never what a node computes. #[wiring] keeps the tokens at the call site, before erasure — and because text and closure come from the same tokens, drift is structurally impossible. That was the failure that sank the deleted legacy codegen retrofit.

The three pieces

quote / emit — func! keeps a closure's tokens; EmitLiteral renders data configs back to source. Configs are never erased (the value sits in the op's Cfg cell), so #[op(emit_cfg)] records them with no annotation at all.

codegen — runs the wiring, walks the graph, prints nitro! input. Refuses rather than emitting a partial artifact, reporting every reason at once.

#[wiring] — the attribute that removes the annotation burden entirely. It rewrites every closure-carrying method call to keep its tokens, and runs free-variable analysis to find captures, so move |p| p - fee needs no declaration.

Decisions worth reviewing

Capture detection is soft, and that is the load-bearing choice. Captures render through emit::Probe, an autoref-specialisation ladder yielding Some(literal) when the type implements EmitLiteral and None when it does not. The attribute annotates a whole wiring function, most of whose nodes are never emitted — so an EmitLiteral bound per detected capture would make orders.map(move |o| book.lock()…) over an Arc<Mutex<Book>> a hard compile error in ordinary wiring, and #[wiring] unusable on any real graph. Instead the wiring compiles and runs; only generation refuses, naming the binding.

Free functions in call position are excluded from detection. Otherwise every closure calling a helper would be mistaken for capturing it. That exclusion turns out to be what makes an ingest graph generatable at all — g.poll(move || venue_feed(venue)) captures a u64, not a receiver. The trade is that a captured callable invoked as f(x) is missed; documented on free_vars with the other two escape classes.

§4's OpFn bound does not work (D28). Binding closure-config ops by a trait with two implementors costs closure signature inference — rustc propagates it only from Fn-family bounds. Measured: ~370 errors across 41 targets, and the residue after reverting the fluent layer was entirely inside nitro! blocks, because compiled() emits closure literals into forwarders whose bounds come from the op. Ops keep Fn and never see a QuotedFn.

A generated _q twin per op was built, then deleted. It was the pre-attribute answer; #[wiring] records the same thing with no call-site annotation, so sixteen extra Stream methods bought nothing. Removed before landing so main never carries a surface we intended to remove.

Bugs found and fixed along the way

• fold/scan silently dropped their seed and reported the graph emittable — Fold's Cfg is its closure, so the seed arrives via #[op(init_arg)] with no other trace on the node. That is the partial emission codegen's own docs promise cannot happen.
• [T; N] and 5-tuples fell off EmitLiteral — quiet gaps before, but capture detection routes through it, so they had become refusals.
• poll was expressible in nitro! but not emittable — #[op(no_builder)] suppressed the set_node_build call recording the method name, so the walker saw a nameless node and refused with advice about variadic ops. Fixing it joins Make busy-poll sources expressible in nitro!, and correct the compiled-tier IO classification #758 to the generator: a busy-poll ingest graph whose shape comes from run-time config now compiles.

Operational guards

• check_artifact — staleness is the failure with no runtime symptom (an artifact built from last quarter's fees compiles, runs, looks plausible). One comparison in a test catches both a config change nobody regenerated for and a hand edit.
• prettyplease formatting — the artifact is the review surface, so it has to be readable. rustfmt does not format inside macro bodies, so the text must arrive right.
• Named nodes (n6_join, not n6) and refusals that name a line, via #[track_caller] on every generated fluent method — including the node nothing annotated, which is exactly the one a refusal exists for.

How it was verified

• cargo fmt --all
• cargo lint and cargo lint-all — both green
• cargo test --manifest-path crates/wingfoil/Cargo.toml --all-features — 91 suites, plus 15 doctests, 5 derive unit tests, scripts/check-example-docs.sh at 48 targets. The nine *_integration suites fail on Socket not found: /var/run/docker.sock — testcontainers, no Docker in this sandbox, unrelated to this diff.
• New behaviour covered by tests asserting values and tick times

How correctness is established, since emission cannot be proven by inspection: every expected artifact sits in the test file as a real nitro! block and the walker must emit byte-identical text — strings match plus the file compiles means the output is valid wiring source by construction, not merely plausible. Parity then runs against the graph it came from, across interpreted, nested and compiled, on values and tick times. EmitLiteral round-trips the same way.

Two mechanisms were spiked before anything was built on them, rather than assumed: the Probe ladder (the double && is load-bearing — with one, the fallback always wins), and #[track_caller] on a trait impl method (no declaration attribute needed, and it survives macro_rules! generation).

Try it

cargo run --manifest-path crates/wingfoil/Cargo.toml --example codegen

Runs both passes in one binary — the checked-in artifacts are include!d — and prints a refusal at the end so the failure mode is visible too.

Known limits, each pinned by a test

Variadic ops (merge_all, combine) carry no build name and are refused. feedback is inexpressible — a cycle is not straight-line emission. Three capture-detection escape classes, all failing as pass-2 compile errors rather than wrong graphs. HashMap deliberately has no EmitLiteral impl: its iteration order varies run to run, so an artifact built from one would churn between identical generations.

Wake-driven I/O (external/channel) is still excluded from compiled(), tracked as #502/#503 and C4. Busy-poll ingest — the shape a compiled graph actually wants, since kernel-bypass NICs are polled rather than woken — works end to end. Closing the rest needs nitro! to express a tuple-returning source and a scoped entry point so a producer handle can escape before the run starts; that is a project, not a follow-up, and deliberately not in here.

---

Generated by Claude Code