BLOCKING: specify the merkle node id as canonical JSON (RFC 8785), JSON types only

[vincenzoml]

Blocking. Node identity is the one thing in this system that must mean the
same on every machine, in every process, in every language a future
implementation is written in — it is the key of the results store, the basis of
CSE, and the whole premise of a cache shared between users. Today it is defined
by an ad-hoc incremental encoder in lazy/hash.py::_feed, with per-type tags
(i, f, s, [, {, o) invented for the purpose and a final else that
falls back to str(value). That encoder is a Python implementation detail
standing where a specification belongs.

The direction, agreed

Define the hashed pre-image as canonical JSON containing only JSON types,
and hash its UTF-8 bytes. The merkle structure stays exactly as it is: a node's
pre-image names its children BY THEIR HASHES, never by their values, so

id = SHA256( canonical_json( { "kind": ..., "operator": "fully.qualified.name",
                               "args": [ id_1, ..., id_n ], ... } ) )

Only JSON types in the pre-image is the load-bearing half: it makes issue #62
(a numpy value changes a node's identity) impossible by construction rather
than by defensive conversion, because a np.float64 is not a JSON type and
cannot enter the pre-image at all.

Three amendments before this is a spec

1. Name the canonicalization. "Standard JSON" is not one. json.dumps is
not canonical: key order, whitespace, escaping and number formatting are all
free. The existing standard is RFC 8785, JSON Canonicalization Scheme (JCS)
— it fixes key ordering (by UTF-16 code units), string escaping, and number
serialization (ECMAScript Number::toString, i.e. shortest round-tripping
form). Referencing an RFC is what makes a second implementation possible;
"sorted keys, no spaces" is what makes it look possible.

2. Decide the number question explicitly, because JSON has one number type
and it is a double. Two consequences, both behaviour changes:

• 1 and 1.0 canonicalize to the same JSON number, so an integer and a float
  of equal value would become the SAME node. Today the i and f tags keep
  them apart. Merging them is arguably right (same value, same node) but it is a
  semantic decision about the language, not a serialization detail.
• Integers above 2^53 are not exactly representable. Today _feed writes an int
  as its decimal digits, so a big integer hashes exactly.

My position: refuse rather than round. The canonical form should accept
integers in the exactly-representable range and reject anything outside it, and
reject NaN and both infinities (which JSON has no syntax for), with an error
naming the node and the attribute. A hash that silently rounds its input is a
hash that silently collides.

3. Keep the whole field set, or justify each omission. The proposed shape
{operator, args} is missing what currently participates:

Field	Why it cannot simply be dropped
attrs	This is where a constant's VALUE lives (kind="constant", attrs={"value": v}) — the leaves of the merkle tree. Drop it and every constant collides with every other. It is also where an operator's parameters live (fold's operator=, a loop's offset=)
kind	constant / primitive / closure are different things with the same operator string
kwargs	Named arguments; same role as args
output_kind	Distinguishes a scalar from a sequence result

So the pre-image is
{"kind", "operator", "args", "kwargs", "attrs", "output_kind"} — and attrs
is precisely the place where "only JSON types" does the work.

What already helps

• Closure bodies are already strings: parser.py's to_syntax() returns str,
  and reducer.py puts function_value.expression.to_syntax() into attrs. So
  the one genuinely non-JSON thing in there is already reduced to a JSON type by
  a declared mapping. That mapping becomes part of the spec.
• The store is already designed for this: the node table is keyed by the raw
  32-byte hash rather than hex or a rowid, explicitly "so ids must be identical
  on every machine and merging must be a plain INSERT OR IGNORE".

What it costs, and the one trap

_feed exists for a measured reason, stated in its own docstring: the previous
json.dumps(sort_keys=True) per node "rebuilt and re-serialized the whole
payload per node, which dominated startup on full-dataset runs". Plans here are
100k+ nodes and elsewhere millions, so a naive per-node JCS serialization will
regress planning time.

The resolution is to keep the fast path but make it subordinate: the spec is
the canonical JSON; the streaming encoder is an optimization that must be proven
byte-identical to it. That means a test that, for a corpus of nodes covering
every field and every type, asserts stream_encode(node) == jcs(pre_image(node))
against an independent JCS implementation — not against our own second opinion.

Migration, which is the expensive part

Changing the pre-image changes every id, which invalidates every payload in
every store — currently ~800 GB here, plus whatever is on other machines.
STORE_SCHEMA_VERSION (now 5) and the existing reset branch already handle
"drop the payloads, keep the node table", so the mechanism exists. The
decision to make is whether old stores are re-keyed (possible in principle: the
node table holds the full DAG, so ids can be recomputed bottom-up and payload
files relinked) or simply discarded. Re-keying is a day of work and saves a week
of recomputation; discarding is one line and throws away the trained-model
predictions among other things.

Acceptance

1. doc/specs/store/merkle-id.md states the pre-image, the canonicalization by
   RFC 8785 reference, the number rules, and the to_syntax mapping — with a
   worked example: a constant, a two-argument primitive, and a closure, each
   with its pre-image bytes and resulting hex id.
2. A test vector file that a non-Python implementation can be checked against.
3. _feed proven byte-identical to the spec on that corpus.
4. A value that cannot be canonicalized raises at plan time, naming the node
   and the field — closing issue A numpy value in a node's attributes changes its identity: one hashing seam, not a conversion per site #62 by construction.