Wingfoil is a blazingly fast stream processing engine for latency-critical systems: electronic trading, real-time decisioning and streaming ML features.
Wire a graph of calculations once and Wingfoil runs it — interpreted, compiled into a single monomorphized function, or as compiled islands inside an interpreted graph. Backtest it over history, then run it live without changing the wiring.
It ships with production-ready adapters covering tick stores, message buses, market protocols and observability backends, so graphs plug into real data sources and sinks in a line.
9.0 replaces the engine. Coming from 8.x, start with the release notes and the migration guide.
Wire the graph in Rust or Python — the same engine underneath, the same
combinator surface — and stream it to a browser over the web adapter.
| Install | Package | Docs | Source | |
|---|---|---|---|---|
| Rust | cargo add wingfoil |
crates.io | docs.rs | crates/wingfoil/ |
| Python | pip install wingfoil |
PyPI | readthedocs | crates/wingfoil-python/ |
| TypeScript | npm install @wingfoil/client |
npm | js/README.md |
js/ |
Rust is the engine itself — all three Nitro execution tiers,
every op and adapter, and #[op] to add your own. Python gets the
same graph model, combinators and adapters in the wheel, with nodes written in
Python and results out as a pandas frame. TypeScript is a browser client for
the web adapter, decoding the wire
format with the server's own code compiled to wasm. The Python wheel and the
browser client both track the engine version, so one number covers all three
registries.
- Fast: ~27 ns of engine overhead per node cycle, from a topologically sorted DAG execution engine that visits each node once per tick.
- Nitro — three execution tiers, one wiring: interpreted, compiled, or a compiled island — all derived from the same definition, so they cannot drift. Compiled runs 4.4×–37× faster.
- Backtesting: replay historical data deterministically off source-driven engine time, then run the identical graph live. Same-instant values ride a single burst — never coalesced, never latest-wins, never dropped, in either mode.
- Adapters: production-ready integrations for KDB+, PostgreSQL, Kafka, Redis, Fluvio, etcd, ZeroMQ, FIX 4.4, iceoryx2, Aeron, WebSocket, Prometheus, OpenTelemetry, CSV, augurs and more — one runnable example each.
- Latency tracing: per-hop wall-clock stamps aggregating into one report, across shared memory and the wire.
- Multi-language: a Rust crate, a Python package and a TypeScript client — one line each to install.
- Graph dynamism: add and remove nodes on a running graph, between cycles.
- Async/Tokio: seamless integration at your graph edges.
- Multi-threading: distribute graph execution across cores, with no locks on the graph execution path.
- Errors are values: every lifecycle function returns a
Result; a producer error propagates into the graph and aborts the run with context. - Extensible without forking: sources, combinators, statistics and adapters
are extension traits, and your own ops get interpreted and compiled coverage
from
#[op].
A simple linear pipeline, with all nodes ticking in lock-step:
use std::time::Duration;
use wingfoil::{RunFor, RunMode};
use wingfoil::prelude::*;
fn main() {
GraphBuilder::new()
.ticker(Duration::from_secs(1))
.count()
.map(|i| format!("hello, world {i}"))
.print()
.build()
.run(RunMode::RealTime, RunFor::Cycles(3))
.unwrap();
}The same graph from Python — run defaults to deterministic historical replay,
so this one finishes instantly rather than taking three seconds:
import wingfoil as wf
g = wf.Graph()
(
g.counter(period_nanos=1_000_000_000) # tick every second: 1, 2, 3, …
.map(lambda n: f"hello, world {n}")
.print() # print each value, pass it through
)
g.run(cycles=3)Either way, this output is produced:
hello, world 1
hello, world 2
hello, world 3
Wingfoil lets you wire up complex business logic, splitting and recombining streams and modulating the frequency of data. Adapters make it easy to plug in real data sources and sinks. Here we load a CSV of AAPL limit orders, maintain an order book with the lobster crate, derive trades and two-way prices, and export both back to CSV:
let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);
let g = GraphBuilder::new();
let (fills, prices) = csv_read(&g, &source_path, get_time, true, None)?
.map(move |chunk: &Burst<Message>| process_orders(chunk, &book))
.split();
let _prices_sink = prices.filter_none().distinct().csv_write(&prices_path)?;
let _fills_sink = fills.csv_write(&fills_path)?;
g.build().run(RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)?;The frequencies of the inputs and outputs are all different to each other — messages arrive in same-timestamp bursts, the top of book changes less often, and trades are sparser still. This output is produced:
An hour of market data — 91,998 messages — replays in about a tenth of a second. Full example.
Nitro is the tier system: one wiring function, wrapped in
nitro! { fn my_graph(g: &GraphBuilder) -> ... }, expands to a module offering
all three tiers:
| Tier | Entry point | What it is |
|---|---|---|
| Interpreted | fluent chaining directly, or my_graph::interpreted() |
One dyn boundary per op; open world — threaded/busy-poll sources, feedback, bursts. |
| Compiled | my_graph::compiled(run_mode, run_for) |
The whole graph monomorphized into one function, state in locals — fastest, static DAGs. |
| Nested (island) | my_graph::nested(&g, inputs...) |
A compiled sub-graph mounted as one node of an interpreted graph — hot core compiled, edges stay open. |
Semantics live once, in each op's cycle function — the tiers differ only in
how the engine reaches it, so there is no duplicated execution logic behind
those three doors. core/dual_mode
has the rules governing what a nitro! wiring accepts.
Read the ratios, not the absolute times: these were captured on shared
4-core cloud VMs, each comparison measured back to back in the same run. Full
method, caveats and per-workload tables:
benches/README.md.
| Measurement | |
|---|---|
| Engine overhead per node cycle | ~27 ns (10×10 graph, 100 nodes, every node ticking every cycle) |
| Compiled vs interpreted | 4.4×–37× faster across eight workloads |
| Nested island vs interpreted | 2.2×–10.2× faster |
| Interpreted vs the legacy engine | 0.56×–0.84× — the port is faster on all eight |
| vs rxrust / tokio async streams | ~79× / ~134× faster at depth 10, and the gap grows with depth |
Wingfoil visits every node once per tick, in topological order. Libraries that
propagate along one path at a time re-visit shared nodes once per path — so on a
branch-and-recombine graph their cost doubles with every level while Wingfoil's
stays flat. core/topological_sort
explains the mechanism in 40 lines.
Where the engine sits against FPGA, kernel-bypass and GC'd stacks — and what is deliberately not claimed — is in where wingfoil currently sits, which ends with the four projects that move that line. All four are open: see Get Involved.
44 runnable examples, each in its own directory with a README covering what it
teaches, the wiring, and its expected output. Full index:
examples/README.md.
If you are new, run these three in order — they cover the whole model between them:
cargo run -p wingfoil --example hello_graph # wire → build → run
cargo run -p wingfoil --example ema_crossover # fold/join/map/filter at backtest scale
cargo run -p wingfoil --features csv --example order_bookNo services, no feature flags — these run with a plain cargo run.
| Example | Description |
|---|---|
hello_graph |
The smallest complete program: wire, build, run. |
ema_crossover |
A backtest-shaped graph — fold, join, map and filter over a price series. |
order_book |
Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, write both back out. |
run_mode |
Swap RunMode::RealTime and RunMode::HistoricalFrom over the same wiring, for backtesting. |
dual_mode |
One wiring, three execution tiers — interpreted, compiled, and a compiled island — proven to agree. |
topological_sort |
Why topologically sorted execution avoids the O(2^N) node explosion of naive per-path propagation. |
dynamism |
Add and remove nodes on a running graph — one price book, four wirings. |
feedback |
Close a loop between two nodes with feedback — a proportional control loop a plain DAG cannot express. |
statistics |
Streaming statistics: EWMA, cumulative and rolling mean/variance/std/min/max/median, over sample- and time-based windows. |
async |
Tokio async/await at the graph's edges, with the core graph staying synchronous. |
async_source |
An async quote feed driving the graph through an external source. |
threading |
Distribute graph execution across worker threads, with no locks on the execution path. |
spawn |
Offload slow work off the graph thread with spawn / spawn_map. |
tracing |
Observability: the logged debug tap and the engine's own spans. |
introspect |
Read back the graph you wired — text, Mermaid, DOT, JSON or GML. |
One directory per adapter, each behind its cargo feature. See each README for the service to start and the command to run.
| Example | Description |
|---|---|
kdb |
KDB+ in three parts: time-sliced reads, LRU-cached reads, and a round-trip write/read/validate. |
postgres |
PostgreSQL — time-sliced historical reads and streaming writes, round-tripped and asserted to tie out. |
kafka |
Consume a Kafka topic, transform each record, produce to another. |
fluvio |
Fluvio — seed a topic, consume it, transform, write to a second topic, from one GraphBuilder. |
redis |
Redis Pub/Sub end to end: publish, subscribe, transform, republish. |
etcd |
Watch an etcd key prefix, transform the values, write them back under another. |
zmq |
ZeroMQ pub/sub, with direct addressing or etcd service discovery. |
fix |
FIX 4.4 — an acceptor and an initiator in one process, over a loopback session. |
iceoryx2 |
Zero-copy IPC over shared memory, in spin, threaded and signaled polling modes. |
aeron |
Low-latency Aeron UDP/IPC transport — publish and subscribe over aeron:ipc. |
web |
Stream a synthetic mid-price to a browser over WebSocket, and take UI events back in. |
ws |
A reconnecting WebSocket client feeding a graph — the transport half of a venue adapter. |
prometheus |
Serve GET /metrics in the Prometheus text format for a scraper or Grafana. |
otlp |
Push stream values to an OpenTelemetry backend over OTLP. |
telemetry |
The two exporters side by side — pull-based scraping vs push — with a Grafana stack. |
csv |
Replay a CSV as a deterministic historical burst stream, transform it, write it back. The one to read first — it needs no server. |
lines |
Line-oriented files in both directions — the smallest complete I/O edge. |
augurs |
On-graph time-series analysis with Grafana's augurs: forecasting, outliers, changepoints, seasonality, DTW, clustering. |
| Example | Description |
|---|---|
latency |
A two-process pipeline over iceoryx2 with per-hop stamping and an end-of-run report. |
trading_e2e |
Browser to live venue and back: WebSocket in, shared memory across processes, FIX/TLS out, with Grafana dashboards over the whole path. |
- Explore the examples
- Read the release notes
- Compare the field: stream processing, dataflow and trading frameworks
- Browse the crates
- Read the benchmarks
- Use it from Python:
wingfoil-python - Use it from the browser:
@wingfoil/client - See CONTRIBUTING to build, test and contribute
Four projects are open, and none of them has landed. Each is separable enough to be carried end to end by one person, and each links to a design rather than starting from a blank page:
| Project | What it moves | Where it stands |
|---|---|---|
| Core pin — pin the graph thread to an isolated core, with the NUMA and warm-up knobs beside it | Deployment discipline — the dominant end-to-end win in the showcase deployment | #392. A working Linux implementation already sits in examples/showcase/trading_e2e/shared.rs; the job is promoting it into runtime/ |
| Kernel bypass — Onload validation, then a raw ef_vi/DPDK source | Ingress, and the wire-to-trade number the benchmarks currently decline to claim | Items 1 and 7 of the trading roadmap. The first rung needs a Solarflare NIC and a measurement run, not a diff |
| Project Lightning — compiled graphs generated from procedurally wired ones | Config-driven topologies onto Nitro's compiled tier, where nitro! structurally cannot follow |
#726, implemented on #769 — open and unmerged, so none of it is on main yet |
| Project Metal — FPGA/Verilog emission (RHDL) behind that same front-end | The sub-microsecond class: the graph as gateware, with the backtest as its testbench | #727 — exploratory, gated behind Lightning on a hand-written de-risk spike |
The fuller picture, with what each one is worth against measured numbers, is in what moves the line.
We want to hear from you! Especially if you:
- are interested in contributing
- know of a project that Wingfoil would be well-suited for
- would like to request a feature or report a bug
- have any feedback
Please do get in touch:
- ping us on discord
- email us at hello@wingfoil.io
- submit an issue
- get involved in the discussion