Context
simplexity is deprecating simplexity/generative_processes in favour of processes vendored from
this repo. Run management can now instantiate a vendored process on equal terms with a simplexity one
(Astera-org/simplexity#198), and full deprecation is gated on this repo reaching feature parity
with that package.
This issue tracks what "parity" is currently missing. Measured against the 21 modules / 3,754 lines
in simplexity/generative_processes at simplexity 4075750.
The good news first: for a base GHMM the operational delta is small. Of the eight members of
simplexity's process interface, a vendored ghmm/process.py plus a ~50-line adapter satisfies six
directly — obs_dist, sample, update, seq_prob, init's eta_0, and Ts.shape[0] map onto
them one-for-one. A vendored process already trains end-to-end under simplexity's runner.
1. Log-space operations
Missing: log_observation_probability_distribution(log_belief_state) and
log_probability(observations).
These are the only two interface members with no counterpart here. They are not on the training or
generation path — simplexity's generation helpers never call them — but belief-state analysis needs
them, because log space is what keeps long-sequence probabilities numerically stable.
simplexity has modelled this as an optional extension protocol rather than a core requirement, so this
does not block vendored processes from training. It does block belief-state analysis on one.
2. Framing tokens and batched generation
SPEC.md §6.2 specifies BOS/EOS/PAD augmentation and §6.3 requires batched generation, but neither is
implemented — generate produces a single unaugmented sequence. §7.3 explicitly excludes generation
from numerical conformance, so this is specified-but-untested territory.
Consumers currently implement both in their adapter. That is arguably correct (it is workload-shaped,
exactly the sort of thing USAGE.md says to adapt), but since the spec does specify the layout it
is worth deciding whether a reference implementation belongs here.
Relatedly, Appendix A marks device placement and DLPack interop explicitly optional and outside
conformance, so the torch bridge equivalent (torch_generator.py) is clearly consumer-side. Noting it
only so it is not mistaken for a gap.
3. Process-zoo breadth
transition_matrices/ here covers zero_one_random, cycle, mess, bloch_walk, moon plus
expand_vocab / compress_vocab. simplexity has 16 families, so 11 are absent: fanizza,
tom_quantum, post_quantum, rrxor, leaky_rrxor, days_of_week, even_ones, matching_parens,
no_consecutive_ones, mr_name, sns, coin.
Whether this counts toward parity is genuinely open, because USAGE.md says "Process definitions are
your data, not our code — the repo does not own a process zoo" while transition_matrices/ exists
anyway. Tracked from the other side in Astera-org/simplexity#195; my lean there is that these stay in
simplexity as data, in which case they should be struck from the parity definition rather than
upstreamed.
4. Mixed-state presentation — probably not parity
Belief-tree enumeration (MixedStateTree, myopic entropy, 576 lines) has no counterpart here, and
arguably should not: it is analysis over a process rather than an operation of one, and the spec
covers belief updates rather than tree enumeration. Being decided in
Astera-org/simplexity#194.
Flagging it here because if it is considered parity, the gate cannot open until this repo grows a
substantial analysis surface — which would cut against "modules, not a package".
What would help most
A decision on which of 1–4 count toward the gate. Items 3 and 4 look like scope questions rather than
work; item 1 is small and concrete; item 2 is a judgement call about how much assembly consumers
should do.
Separately and independently: the conformance vectors (#8) matter more than any single item here,
because they are what makes vendoring safe at all.
Context
simplexityis deprecatingsimplexity/generative_processesin favour of processes vendored fromthis repo. Run management can now instantiate a vendored process on equal terms with a simplexity one
(Astera-org/simplexity#198), and full deprecation is gated on this repo reaching feature parity
with that package.
This issue tracks what "parity" is currently missing. Measured against the 21 modules / 3,754 lines
in
simplexity/generative_processesat simplexity4075750.The good news first: for a base GHMM the operational delta is small. Of the eight members of
simplexity's process interface, a vendored
ghmm/process.pyplus a ~50-line adapter satisfies sixdirectly —
obs_dist,sample,update,seq_prob,init'seta_0, andTs.shape[0]map ontothem one-for-one. A vendored process already trains end-to-end under simplexity's runner.
1. Log-space operations
Missing:
log_observation_probability_distribution(log_belief_state)andlog_probability(observations).These are the only two interface members with no counterpart here. They are not on the training or
generation path — simplexity's generation helpers never call them — but belief-state analysis needs
them, because log space is what keeps long-sequence probabilities numerically stable.
simplexity has modelled this as an optional extension protocol rather than a core requirement, so this
does not block vendored processes from training. It does block belief-state analysis on one.
2. Framing tokens and batched generation
SPEC.md§6.2 specifies BOS/EOS/PAD augmentation and §6.3 requires batched generation, but neither isimplemented —
generateproduces a single unaugmented sequence. §7.3 explicitly excludes generationfrom numerical conformance, so this is specified-but-untested territory.
Consumers currently implement both in their adapter. That is arguably correct (it is workload-shaped,
exactly the sort of thing
USAGE.mdsays to adapt), but since the spec does specify the layout itis worth deciding whether a reference implementation belongs here.
Relatedly,
Appendix Amarks device placement and DLPack interop explicitly optional and outsideconformance, so the torch bridge equivalent (
torch_generator.py) is clearly consumer-side. Noting itonly so it is not mistaken for a gap.
3. Process-zoo breadth
transition_matrices/here coverszero_one_random,cycle,mess,bloch_walk,moonplusexpand_vocab/compress_vocab. simplexity has 16 families, so 11 are absent:fanizza,tom_quantum,post_quantum,rrxor,leaky_rrxor,days_of_week,even_ones,matching_parens,no_consecutive_ones,mr_name,sns,coin.Whether this counts toward parity is genuinely open, because
USAGE.mdsays "Process definitions areyour data, not our code — the repo does not own a process zoo" while
transition_matrices/existsanyway. Tracked from the other side in Astera-org/simplexity#195; my lean there is that these stay in
simplexity as data, in which case they should be struck from the parity definition rather than
upstreamed.
4. Mixed-state presentation — probably not parity
Belief-tree enumeration (
MixedStateTree, myopic entropy, 576 lines) has no counterpart here, andarguably should not: it is analysis over a process rather than an operation of one, and the spec
covers belief updates rather than tree enumeration. Being decided in
Astera-org/simplexity#194.
Flagging it here because if it is considered parity, the gate cannot open until this repo grows a
substantial analysis surface — which would cut against "modules, not a package".
What would help most
A decision on which of 1–4 count toward the gate. Items 3 and 4 look like scope questions rather than
work; item 1 is small and concrete; item 2 is a judgement call about how much assembly consumers
should do.
Separately and independently: the conformance vectors (#8) matter more than any single item here,
because they are what makes vendoring safe at all.