Skip to content

Capability gaps gating simplexity's deprecation of generative_processes #9

Description

@ealt

Context

simplexity is deprecating simplexity/generative_processes in favour of processes vendored from
this repo. Run management can now instantiate a vendored process on equal terms with a simplexity one
(Astera-org/simplexity#198), and full deprecation is gated on this repo reaching feature parity
with that package.

This issue tracks what "parity" is currently missing. Measured against the 21 modules / 3,754 lines
in simplexity/generative_processes at simplexity 4075750.

The good news first: for a base GHMM the operational delta is small. Of the eight members of
simplexity's process interface, a vendored ghmm/process.py plus a ~50-line adapter satisfies six
directly — obs_dist, sample, update, seq_prob, init's eta_0, and Ts.shape[0] map onto
them one-for-one. A vendored process already trains end-to-end under simplexity's runner.

1. Log-space operations

Missing: log_observation_probability_distribution(log_belief_state) and
log_probability(observations).

These are the only two interface members with no counterpart here. They are not on the training or
generation path — simplexity's generation helpers never call them — but belief-state analysis needs
them, because log space is what keeps long-sequence probabilities numerically stable.

simplexity has modelled this as an optional extension protocol rather than a core requirement, so this
does not block vendored processes from training. It does block belief-state analysis on one.

2. Framing tokens and batched generation

SPEC.md §6.2 specifies BOS/EOS/PAD augmentation and §6.3 requires batched generation, but neither is
implemented — generate produces a single unaugmented sequence. §7.3 explicitly excludes generation
from numerical conformance, so this is specified-but-untested territory.

Consumers currently implement both in their adapter. That is arguably correct (it is workload-shaped,
exactly the sort of thing USAGE.md says to adapt), but since the spec does specify the layout it
is worth deciding whether a reference implementation belongs here.

Relatedly, Appendix A marks device placement and DLPack interop explicitly optional and outside
conformance, so the torch bridge equivalent (torch_generator.py) is clearly consumer-side. Noting it
only so it is not mistaken for a gap.

3. Process-zoo breadth

transition_matrices/ here covers zero_one_random, cycle, mess, bloch_walk, moon plus
expand_vocab / compress_vocab. simplexity has 16 families, so 11 are absent: fanizza,
tom_quantum, post_quantum, rrxor, leaky_rrxor, days_of_week, even_ones, matching_parens,
no_consecutive_ones, mr_name, sns, coin.

Whether this counts toward parity is genuinely open, because USAGE.md says "Process definitions are
your data, not our code — the repo does not own a process zoo" while transition_matrices/ exists
anyway. Tracked from the other side in Astera-org/simplexity#195; my lean there is that these stay in
simplexity as data, in which case they should be struck from the parity definition rather than
upstreamed.

4. Mixed-state presentation — probably not parity

Belief-tree enumeration (MixedStateTree, myopic entropy, 576 lines) has no counterpart here, and
arguably should not: it is analysis over a process rather than an operation of one, and the spec
covers belief updates rather than tree enumeration. Being decided in
Astera-org/simplexity#194.

Flagging it here because if it is considered parity, the gate cannot open until this repo grows a
substantial analysis surface — which would cut against "modules, not a package".

What would help most

A decision on which of 1–4 count toward the gate. Items 3 and 4 look like scope questions rather than
work; item 1 is small and concrete; item 2 is a judgement call about how much assembly consumers
should do.

Separately and independently: the conformance vectors (#8) matter more than any single item here,
because they are what makes vendoring safe at all.

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions