scarlet/wire: type-directed encode/decode of values to bytes

[alii]

Status: design agreed with @alii on 2026-08-13, not started. This issue is the implementation plan; nothing below is built. Line references are against 97e4c70.

What this is

A stdlib module scarlet/wire that turns a value into bytes and back:

import scarlet/wire

type Event {
	Joined(user String, at Int)
	Said(user String, text String, tags Array(String))
}

pub fn main() {
	bytes = wire.encode(Said('ali', 'hi', ['a']))        // Binary
	match wire.decode(bytes) {                            // Result(Event, wire.DecodeError)
		Ok(Said(user, text, _)) -> println('${user}: ${text}')
		Ok(Joined(..)) -> Nil
		Err(e) -> println(string.inspect(e))
	}
}

It is Erlang's term_to_binary / binary_to_term, with two differences that are the whole point:

1. decode is type-safe. Ok(x) means x has exactly the type the call site was inferred at. There is no dynamic value type and no runtime type test in user code; bad bytes are an Err, never a wrong-shaped value and never a crash.
2. encode is total. A type that has no byte representation (one containing a fn, Subject, Pid, Socket, Port, or an opaque type from another module) is a compile error at the encode/decode call, so encode returns a plain Binary.

Both follow from one mechanism: the compiler knows the concrete type at every wire.encode/wire.decode call, so it emits a descriptor of that type into the program and hands it to the VM op. Encoding walks the value under the descriptor's guidance; decoding builds the value from the descriptor, checking the bytes against it as it goes. Nothing is inferred from the bytes.

Naming: the word is "value", not "term" (this repo has never used "term"), and the module is named for the format the way scarlet/json is, so it reads wire.encode / wire.decode next to json.encode / json.parse. examples/wire_format.scrl is a bit-syntax example and is unrelated; it should be renamed (bit_syntax.scrl) when this lands so the two do not get confused.

Why now

Today a value can only leave a program as text: string.inspect (not parseable back), json.encode (only a hand-built Json tree, and reading it back is a hand-written decoder per type via scarlet/json/decode). Everything the process model is heading towards needs values as bytes with no per-type code: sending a message to a process in another OS process over a port (scarlet/os/port) or a socket, an ETS-style table or on-disk cache (Binary in, same type out), and eventually distribution. Each of those wants exactly "bytes for a value of type t, and t back out, checked", which is what a type-directed codec is and what a hand-written decoder per type is not.

Scope

Decided here: the public API, which types are encodable, the descriptor, the byte format, the compiler and VM changes, and the tests that pin the guarantees. Not decided here, and not needed for v1: schema evolution (decoding bytes written by an older version of a type), opting opaque types in from outside their module, and using descriptors to drive json.decode into a type. Those are listed under Open questions with the shape I would expect them to take, so v1 does not paint them out.

Public API

crates/scarlet_core/src/std/scarlet/wire.scrl:

// A value of any encodable type as bytes. Total: a type this cannot encode
// is rejected when the program is compiled.
@vm(wire__encode)
pub fn encode(value a) Binary

// The value `bytes` holds, as the type this call is used at. The type must
// be known where `decode` is called; `Ok` is a well-formed value of it.
@vm(wire__decode)
pub fn decode(bytes Binary) Result(a, DecodeError)

pub type DecodeError {
	// Fewer bytes than the value needs. `needed` is how many more, when known.
	Truncated
	// Not produced by `wire.encode` (bad magic), or by a format version this
	// runtime does not read.
	NotWire
	// Encoded from a type of a different shape. Both fingerprints, so a log
	// line can say which side changed.
	SchemaMismatch(expected Int, found Int)
	// Well-framed bytes that do not hold a value of the type: a variant tag
	// out of range, a string that is not UTF-8, a count larger than the
	// remaining input. `offset` is the byte it went wrong at.
	Malformed(offset Int, what String)
	// A complete value followed by more bytes. Reported rather than ignored,
	// because a caller that framed its own messages has lost sync.
	TrailingBytes(count Int)
}

The signatures are ordinary HM signatures — a is a real type variable — and the type-directed part is invisible in the source: the restriction that a be concrete and encodable is checked by the compiler at each call, the same way exhaustiveness is checked at each match. There is no attribute on the user's types and nothing to derive. This is why the design does not need typeclasses: the information is per call site, not per type.

Which types are encodable

The descriptor builder is a total function from a resolved type (ResolvedNode) to either a descriptor or a diagnostic:

Type	Encodable	Notes
Int, Float, String, Binary	yes	the four bodiless prelude types with a representation
Array(t)	if t is	a Range value has type Array(Int) and encodes as its elements
Map(k, v)	if k and v are	the one bodiless stdlib type on the list; needs a prelude-style binding so the builder can recognise its TypeId
tuples	if every element is
any type with a body (Bool, Nil, Option, Result, Order, every user enum and record)	if every field of every variant is, and its constructors are visible at the call site	generic types are encoded at their instantiation: Option(Int) and Option(String) get different descriptors; recursive types get a descriptor node that refers back to itself
fn(..)	compile error	no representation
Subject, Pid, Socket, TlsSocket, Port, json.Doc, any other bodiless type	compile error	a handle names something in this OS process; bytes cannot carry it. Distribution, if it happens, would relax Subject/Pid deliberately, in the runtime, not here
a type variable (decode inside a generic function, or decode(b) whose result is never constrained)	compile error: "the type wire.decode produces here is not known; annotate the binding"	this is the restriction that makes the whole thing work without typeclasses
an opaque type, outside its module	compile error, both directions for v1	see below

The opaque rule is the one judgement call. decode builds values by constructor without running any of the module's code, so decoding an opaque type outside its module is exactly what opaque exists to forbid: decimal.Decimal is {units, scale} and bytes could produce a scale the module never allows. Inside the defining module both directions are fine (the module is already trusted with its own representation), so scarlet/decimal can export to_wire/from_wire if it wants to, and validate. The visibility bit already exists — analysis.rs:845 computes ctors_public = is_public && !opaque to decide what goes in the module interface — it just needs to be recorded on the type so the builder can read it. The cost of this rule is real and worth naming: in v1 a record with a Decimal or Instant field is not encodable outside those modules. Instant should not be (it is a monotonic reading); Decimal is the case the opt-in in Open questions is for. Encoding is forbidden too, rather than just decoding, so that encode and decode accept the same set of types; an asymmetric rule would let a program write bytes it can never read back.

The descriptor

One descriptor per call site's type, built at compile time, stored as a constant, and identified by structure so identical types share one. Shape:

Desc        = table of Node, plus root index, plus fingerprint u64
Node        = Int | Float | String | Binary
            | Array(node) | Map(node, node) | Tuple(nodes…)
            | Data(variants…)                       -- anything with a body
Variant     = (template TemplateIdx, fields: nodes…)

Recursive types work because a field refers to a node by index, so type List(a) { Cons(head a, tail List(a)) End } at List(Int) is a two-node table whose Cons.tail points at node 0. The descriptor is a tree of types, which is finite; the value being encoded is a DAG of arbitrary depth, which is the VM's problem (below).

template is an index into the existing Program.templates table of EnumTemplates, which is how the VM already constructs stdlib values (Ok, Some, NetError, …) for the ABI: type id, names and labels pre-interned in the frozen area, and instantiate(payload) builds the value with no per-value allocation beyond the value itself. Decoding a variant is therefore the same call the ABI makes. Two consequences to plan for: bind_abi currently rebuilds the table from empty (abi.rs:170), so it has to append instead (or run before elaboration); and type ids are per-program, which is fine because the template lives in the program doing the decoding — nothing about type identity is ever written to the bytes.

The fingerprint is a 64-bit hash of the descriptor's structure: node kinds, variant order, constructor names and field labels, computed over the table in index order so a recursive type hashes without recursing. It deliberately includes names, so renaming a constructor is a schema change; it deliberately excludes the type's own name and its module, so two programs that declare the same shape independently interoperate, and moving a type between modules is not a schema change. It is written into every encoded value and checked first when decoding, which is what turns "the type changed" from silent misreads into SchemaMismatch in the first nine bytes.

How the descriptor reaches the op: Atom::PrimOp already carries an Imm::Const immediate for IndexOr's default, emit.rs:257 flattens it to the operand, and the Core IR codec serialises it, so the descriptor is a constant-pool entry (a frozen tuple graph, like any other compound constant) and WireEncode/WireDecode take it as their operand. emit.rs:257 currently allows the Const immediate only on IndexOr and needs the two new ops added.

Byte format

Little-endian throughout. Numbers written by the type, so there are no per-value tags; the descriptor on the reading side says what comes next.

value      := magic "SW" · version u8 (= 1) · fingerprint u64 · body
Int        := zigzag LEB128            (values outside i64 do not exist: Int is 64-bit)
Float      := 8 bytes, f64 bits         (the VM never holds NaN/Inf, so decode maps a
                                         non-finite reading to 0.0, matching arithmetic)
String     := len LEB128 · UTF-8 bytes  (validated on decode)
Binary     := bit_len LEB128 · ceil(bit_len/8) bytes   (bit-level binaries round-trip)
Array      := count LEB128 · elements
Map        := count LEB128 · key value pairs            (decode rebuilds the map; duplicate
                                                          keys are Malformed, since encode
                                                          never writes them)
Tuple      := elements                                   (arity is in the descriptor)
Data       := variant LEB128 · fields                    (omitted entirely when the type has
                                                          one variant, so a record costs
                                                          nothing but its fields, and Nil
                                                          costs zero bytes)

Size for the Said example above: 11 bytes of header, then 3 ali 2 hi 1 1 a = 12 bytes. The header is the price of the schema check; a caller framing many values of one type into a stream can strip it later with an encode_body-style variant if that ever matters, and it should not be designed in now.

Compiler changes

In dependency order. All in scarlet_core unless noted.

1. Record constructor visibility on types. analysis.rs:845 knows it; put it on the type's info so it survives into module interfaces and the static stdlib blob, and add it to the interface codec.
2. A Map binding. The builder recognises prims through pool.prims() and Bool/Binary through PreludeBindings (clif.rs:208 does the same for the JIT); Map needs the same treatment.
3. Descriptor builder — new file typed_ir/wire.rs: ResolvedPool + RTy + type env → Result<Desc, WireRefusal>. Instantiates each variant's field types at the type's arguments (the elaborator already has the substitution machinery for this; the builder should call it, not reimplement it), interns nodes by structure, mints templates, computes the fingerprint. Refusals carry the offending sub-type and the path to it (Event.Said.tags: fn(Int) Int has no wire representation) — a refusal three levels down a record must name the field, or the diagnostic is useless.
4. Elaboration. This is where the diagnostic is raised, because Core IR lowering is total by construction and must stay that way. When elaboration resolves a call whose callee is Builtin(WireEncode | WireDecode), it takes the argument's type (encode) or the Ok payload of the call's result type (decode), runs the builder, and either reports or replaces the callee with the descriptor attached — the cleanest shape is TypedCallee::Builtin growing an immediate (Builtin { op, imm }, Imm::None for every existing op) so lower.rs:735 becomes a one-line change. decode's payload type has to be read after zonking the enclosing function, since it is usually fixed by a later match; that is already the order elaboration works in.
5. Builtins as values. array.map(events, wire.encode) goes through eta.rs, which mints a wrapper per use with the use's instantiated type, so it can attach the descriptor the same way. If that turns out fiddly, v1 may reject it ("call wire.encode directly") — see Open questions.
6. Names and emit. wire__encode/wire__decode in bytecode/mod.rs's builtin table; emit.rs:257 accepting Imm::Const for the two ops; DecodeError's constructors bound as ABI slots (scarlet_vm/src/abi.rs slot enum and slots_for, compiler/abi.rs BINDINGS, the fixture rows in template.rs's tests — the same four places every stdlib error type touches).
7. REPL / incremental. Descriptors are constants of the program being built, so the REPL and IncrementalSession need nothing new; the check-only path (scarlet check, LSP) must still run the builder so the diagnostics appear in editors — i.e. it lives in elaboration, not emission, which step 4 already ensures.

VM changes

scarlet_vm/src/vm/wire.rs, two ops, both bridged to the JIT the way IndexOr with its constant operand already is (native.rs:149):

• WireEncode [value] → Binary, operand = descriptor. Walks value and descriptor together with an explicit stack (the value can be a 100k-element list of lists; nothing here may recurse on the native stack — same rule as the graph copier and equality). Writes into a Vec<u8> and wraps it as a whole-buffer binary. The descriptor is only consulted for structure; the value's own tags are debug-asserted against it, since the compiler guaranteed they match.
• WireDecode [Binary] → Result(t, DecodeError), operand = descriptor. Header first (magic, version, fingerprint), then an explicit-stack builder. The input is untrusted, so: every count is checked against the bytes remaining before anything is allocated (a 4-byte input claiming a 2^40-element array is Malformed, not an allocation), strings are UTF-8 validated, variant tags are range-checked against the descriptor, and maps are rebuilt through the normal insert path so their invariants hold. Values are built with the ordinary constructors (instantiate, tuple_in, the seq builder, the map builder), so a decoded value is indistinguishable from a constructed one — that, plus the descriptor being the type, is the type-safety argument in one sentence.
• Reductions: charge per byte or per node so a large decode yields like a large copy does; both ops are pure and never park.

Tests that pin the guarantees

• tests/programs/wire.scrl + golden: round trips for every row of the table above (including a recursive type, a nested generic, a bit-length binary, a range, a map, Nil, a big int), asserting decode(encode(x)) == Ok(x), plus one printed encoding so the byte format itself is pinned and a format change is a visible golden diff.
• Compile-error tests (type_errors.rs style): fn field, Subject field three levels down (checks the path in the message), unconstrained decode, decode inside a generic fn, Decimal outside its module, Decimal inside a module that defines its own opaque type (allowed).
• Schema tests: encode as A, decode as a structurally identical B declared in another module → Ok; decode as a type with one extra field → SchemaMismatch; the fingerprint of a type is stable across two compiles (dis the constant).
• Robustness (Rust test in crates/scarlet/tests/vm_wire.rs): for each fixture, decode every truncation and every single-byte mutation of its encoding; every result must be Ok or Err, never a crash or a hang, and every Ok must survive string.inspect and == against itself (which walks the whole value and would trip the debug tag guards on anything malformed). This is the test that makes "type-safe" a checked property rather than a claim.
• Scale: a 200k-element array and a 100k-deep cons list round-trip (explicit stacks), and the same inputs with a count field forged to 2^40 fail fast without allocating.
• Perf: encode/decode of the bench_service request record against json.encode of the equivalent tree as a reference point; there is no reason for wire to be slower than json and it should be several times faster since it writes no names.

Alternatives considered

A dynamic value type (wire.decode(bytes) Result(wire.Value, _), then the caller pattern-matches — the shape of json.Doc). Simpler compiler: no type-directed anything. It puts the type check back in user code at every use, which is what json/decode's combinators exist to make bearable, and it means every decode allocates an intermediate. The point of this issue is to not have that layer, so it was not chosen; json keeps it because JSON genuinely is schemaless on the wire.

Self-describing bytes (write constructor and field names into the stream, as term_to_binary does, and let decode match by name). Buys tolerance of field reordering and some evolution, and debuggability of raw dumps. Costs 3–10× the bytes for typical records and a name comparison per field on decode, and it does not remove the need for a descriptor (the reader still needs to know what to build), so it is strictly more machinery. The fingerprint gets the safety half of the benefit for eight bytes; the evolution half is an open question below and can be added as a second format version without changing the API.

Typeclass / derive (@derive(wire) on the type, or a Codec(t) value the user threads through, as Gleam does). Threading is the status quo with json/decode; a derive attribute would work but is per-type opt-in that every message type would carry, and it adds a language feature for something the compiler can decide per call site with the information it already has. If a second type-directed operation appears (a typed json.decode is the obvious candidate) the per-call-site machinery here is exactly what it would reuse, so this is not a one-off.

Runtime-only, using the value's own tags (encode needs no descriptor at all: values are self-tagged). True for encode, and it is tempting to skip the compiler work on that side. Rejected because encode is where the "no fn, no Subject" rule has to be enforced — enforcing it at runtime would make encode return a Result or crash, and the whole reason to do this at compile time is that it does not. Encode also uses the descriptor to omit tags and single-variant markers, which is where the size win comes from.

Tradeoffs being accepted

• The type must be concrete at each call. A generic helper fn save(x a) cannot call wire.encode(x); the caller has to encode first and pass bytes. That is the price of having no typeclasses, and it is the same restriction Rust's serde puts on you spelled differently (T: Serialize); it just surfaces at the call site instead of the signature.
• Bytes are tied to a shape, not evolvable: any change to a type in the stream is a SchemaMismatch. Right for messages between two copies of the same program and for caches; wrong for long-lived storage until an evolution story exists (below).
• Opaque types from other modules are excluded, so a Decimal field blocks encoding a record in v1.
• One more thing the elaborator does per program. Descriptors are interned, so a program with a hundred encode calls on ten types builds ten descriptors; the fingerprint hash is over the descriptor, not the program, so compile-time cost is proportional to the types used, not the call sites.

Open questions

1. Opaque opt-in. The likely shape: a module marks an opaque type as wire-safe by exporting a validator (pub fn wire_check(d Decimal) Bool by convention, or an attribute naming one) which decode calls after constructing; encode needs nothing. Needs deciding before Decimal fields become common in messages; not needed for v1.
2. Evolution. Whether to add a self-describing format version (names in the stream, missing fields → Malformed unless the field is an Option) for storage use. Independent of the API; decide when something needs to read old bytes.
3. Builtins as values (array.map(xs, wire.encode)) — support via eta in v1, or reject with a pointer to fn(x) wire.encode(x). Preference: support it, since eta already has the instantiated type; drop to reject only if it costs more than an hour.
4. Range fidelity. A range encodes as its elements (its type is Array(Int)), so decode(encode(0..1_000_000)) is a million-element array. Correct by the types; whether encode should refuse or warn on a huge range is a judgement call — proposal: do nothing, it is what the type says.
5. Reduction accounting granularity, and whether decode of a very large binary should be able to yield mid-value (proposal: no — charge up front from the byte length, like a large copy).
6. Where the fingerprint algorithm is specified. It becomes a compatibility surface the moment two separately compiled programs talk; it should be documented next to the format in wire.scrl's module doc and pinned by the golden, and changing it means bumping the version byte.

Estimate

Compiler (steps 1–7) about a day; VM half a day; stdlib, tests, example and website page half a day. Steps 1–3 can be reviewed on their own before anything is wired up.

[alii]

On second thoughts - I think ideally everything should be serializable, even functions, pids, etc. There's no reason it shouldn't work. Eventually when we have clustering you'll be able to send these things across the network.

[alii]

Status on scarlet/wire, since you asked for updates here.

Landed on master

• e725ec4 — the descriptor builder (typed_ir/wire.rs): walks a type and produces a Desc.
• 4446754 — scarlet/wire.scrl as a stdlib module, plus DecodeError's ABI slots.
• 1a122f5 — narrowed the builder's public surface. cargo hawk check had been reporting 19 findings on wire.rs across e725ec4 and 4446754 while returning exit code 0, so CI was red for about four hours without anyone reading it as red. 17 items went private, 2 dead accessors were deleted, and the 4 remaining suppressions are per-item with a stated reason.
• 3764839 — TypedCallee::Builtin carries an immediate; emit accepts Imm::Const for the two wire ops, without weakening the exhaustive panic for any other pairing.
• b44cde6 + 964dbfa — the fingerprint, its written spec, and what moves it.

The fingerprint's one load-bearing decision: names fold as text, never as StrId. A StrId is an interning index, so folding it produces a fingerprint that changes between compiles — precisely the boundary the fingerprint exists to survive. Children fold as indices and each node's row folds when the table loop reaches it, which is what lets a recursive type hash without recursing.

Eight planted corruptions each go red on their named test. The counterfactual is the part worth reporting: with the same corruption in place but only the stability test deleted, the suite passes green — so a fingerprint that changed between compiles would have shipped as success, and SchemaMismatch would then fire on types nobody had touched.

On your 02:42 comment — that everything should ideally be serializable, functions and pids included — that agrees with something measured here independently. The body's rationale that fn has no representation is not correct as written, and three further statements in the issue don't survive contact with the code:

1. The encodability table calls Socket, TlsSocket, Port and json.Doc bodiless. All four are Custom types with constructors.
2. DecodeError.Truncated's doc names a needed field the variant does not carry.
3. wire.scrl's module doc is cited as the home of the byte format, and no byte format is written in it — though DecodeError.NotWire already documents behaviour (a "magic") that only a written format could define.

A corrected body is drafted and verified but deliberately not applied: editing your issue text is your call.

The open question your comment sharpens, which nobody has decided. If fn and pid become encodable for clustering, the encodability table is not a fixed partition of types — it is a statement about what is implemented today. Those two need different refusal semantics: a type that can never be encoded and one that cannot be encoded yet should not fail the same way, or a receiving node cannot distinguish "this will never work" from "that peer is too old to know this type". That interacts directly with the fingerprint and SchemaMismatch, and it is step 7's territory.

Every ticket in the breakdown was written before your 02:42 comment and assumes the fixed partition.

[alii]

Correcting the impression my last comment gave, and then the fn/pid question properly, because it turns out to be two unrelated problems wearing one label.

Nothing encodes yet, and that is the plan rather than a breakage

My last comment discussed which types the design excludes, which reads as "wire mostly works, minus some types". It doesn't work at all yet. Both VM ops trap:

pub(super) fn wire_encode(&mut self) -> VmResult<()> {
    Err(VmError::internal(
        "wire.encode has no encoder yet: the op is declared, not implemented",
    ))
}

and build_desc — the descriptor builder — has no caller anywhere outside its own file. wire.encode(42) fails today exactly as surely as wire.encode(some_closure).

The module doc says so deliberately: "The declaration surface only. Both ops are in the ISA, both are reachable from Scarlet, and both trap."

What is built and tested: the type→descriptor walk; the fingerprint with its written spec; wire.scrl with encode/decode at their final signatures; DecodeError's five constructors bound as ABI slots so a renamed constructor is a compile error rather than a mis-built value; and the immediate plumbing that lets a descriptor ride to the op.

So the parts that decide what the bytes mean exist. The part that writes the bytes is step 9.

Next is step 7 — elaboration running the builder at each call site and interning the descriptor as a program constant. Both its dependencies just landed. The reason it has to live in elaboration rather than emission is that scarlet check and the LSP never reach emission, so a refusal raised there would never appear in an editor.

Pids: not a serialisation problem

// The identity of a process. Program-unique and never reused, so a Pid held
// after the process ends is safely dead.
pub type Pid

Program-unique is the whole obstacle. A Pid names a process within one program, so shipping one elsewhere yields a name that means nothing at the far end. Erlang pids survive the wire because the node is part of the identity — (node, id).

The current refusal for this class reads "the type is host-backed and has no representation outside this program", and the phrase "outside this program" is only true because there is one program. Give pid identity a node scope — the clustering work — and encoding it becomes trivial and this refusal disappears on its own rather than being overcome. Nothing in wire needs to change to allow it.

Closures: collides with a decision already made

A closure at runtime is func_idx: i32 plus a flat, untyped array of captures. The descriptor is derived from the static type. And:

fn() { socket.send(s, x) } and fn() { n + 1 } have the same type.

So no call site can know which lambda arrives. That matters because encode is declared total — it returns a plain Binary, not a Result, which is what makes encodability a compile-time property. If closures were encodable, a site typed fn() Nil would have to accept one capturing a socket, and encode would have to fail at runtime.

That is a conflict with the design's central promise, not a missing feature.

Worth stating plainly, since the issue body currently says otherwise: "a function has no wire representation" is false. Erlang's NEW_FUN_EXT and EXPORT_EXT serialise funs perfectly well and carry no code; they fail only when the fun is called on a peer lacking the module. The criterion is reconstructibility, not representability. The code comment in wire.rs already says this; the issue body has not caught up.

Making closures encodable therefore means picking one:

1. encode stops being total for closures and returns a Result, moving the failure to runtime; or
2. the descriptor stops being a function of the static type alone and carries capture layout, meaning the compiler records what each lambda captured.

The decision worth making before step 9

Whether encode stays total. Everything above forks on it, and the refusal path, the fingerprint's contents and DecodeError's shape all depend on the answer. It is far cheaper now than after the encoder exists.

A suggestion, not a decision: pids need no answer here — they are blocked on clustering, not on wire, and become encodable for free when process identity gains a node. Closures can wait for a caller that actually wants one. But the totality question should not wait, because step 9 bakes in an answer either way, and silently.

One thing the current design cannot express, which is the root of both: it has no way to distinguish this can never be encoded from this is not encodable yet. A receiving node cannot tell "this will never work" from "that peer is too old to know this type". Every ticket in the breakdown was written assuming the partition is permanent.