The gap
SPEC.md §7 specifies conformance test vectors in detail — the JSON encoding for process definitions
(§7.1.1), state encoding (§7.1.2), structure definitions (§7.1.3), tolerances (§7.2, rel 1e-6 / abs
1e-12), and 18+ named categories (§7.4: ghmm_belief_update, factored_belief_update,
nonergodic_belief_update, …).
No vector files exist in the repo. No JSON fixtures, and nothing under tests/ references them.
Why this matters more than a normal missing-tests issue
The vectors are load-bearing for this repo's entire distribution argument. USAGE.md says:
The invariant that survives your modifications is not the code — it is the spec. After you have
rewritten a module for your use case, run the conformance vectors against your copy: if they pass,
your fork still implements the same math.
and step 2 of "How to consume" is "Copy the relevant conformance vectors and wire them into your test
suite against your copy."
So copying-with-modification is only safe because of the vectors. Without them, a consumer who
specializes a module — which is the point of the model — has no way to check they preserved the
mathematics. The safety net is currently specified but unavailable.
Concrete consequence downstream
simplexity has just relaxed its generative-process contract from a nominal base-class check to a
structural protocol, so that a vendored process from this repo can be instantiated by its runner
(Astera-org/simplexity#198).
A structural check verifies that the required operations exist. It cannot verify they compute the
right thing: it will happily accept a transition_states with reversed arguments, a vocab_size that
lies, or an obs_dist returning unnormalized values. The conformance vectors are the only mechanism
that would catch that class of error, which makes their absence the weakest link in the chain.
The migration guide currently has to tell consumers to wire up vectors and then admit in the same
breath that they do not exist yet, which is not a great look for the first documented consumer path.
Suggested scope
Enough to be useful before being complete:
- The base GHMM categories first (
ghmm_obs_dist, ghmm_belief_update, ghmm_seq_prob,
stationary_distribution) — that covers the minimal liftable unit (ghmm/process.py + utils.py)
that most consumers will vendor first.
- Factored and nonergodic categories after.
- A loader plus a small runner in
tests/ so the vectors are exercised against this repo's own
reference modules — otherwise the vectors themselves are unverified, and a wrong vector is worse
than a missing one.
Deriving the expected values from the reference implementations makes the vectors a regression
baseline rather than an independent check of the reference code. That is probably the right trade
(they exist to police consumers' forks), but worth doing knowingly, and worth spot-checking a few
values against hand computation.
The gap
SPEC.md§7 specifies conformance test vectors in detail — the JSON encoding for process definitions(§7.1.1), state encoding (§7.1.2), structure definitions (§7.1.3), tolerances (§7.2, rel
1e-6/ abs1e-12), and 18+ named categories (§7.4:ghmm_belief_update,factored_belief_update,nonergodic_belief_update, …).No vector files exist in the repo. No JSON fixtures, and nothing under
tests/references them.Why this matters more than a normal missing-tests issue
The vectors are load-bearing for this repo's entire distribution argument.
USAGE.mdsays:and step 2 of "How to consume" is "Copy the relevant conformance vectors and wire them into your test
suite against your copy."
So copying-with-modification is only safe because of the vectors. Without them, a consumer who
specializes a module — which is the point of the model — has no way to check they preserved the
mathematics. The safety net is currently specified but unavailable.
Concrete consequence downstream
simplexityhas just relaxed its generative-process contract from a nominal base-class check to astructural protocol, so that a vendored process from this repo can be instantiated by its runner
(Astera-org/simplexity#198).
A structural check verifies that the required operations exist. It cannot verify they compute the
right thing: it will happily accept a
transition_stateswith reversed arguments, avocab_sizethatlies, or an
obs_distreturning unnormalized values. The conformance vectors are the only mechanismthat would catch that class of error, which makes their absence the weakest link in the chain.
The migration guide currently has to tell consumers to wire up vectors and then admit in the same
breath that they do not exist yet, which is not a great look for the first documented consumer path.
Suggested scope
Enough to be useful before being complete:
ghmm_obs_dist,ghmm_belief_update,ghmm_seq_prob,stationary_distribution) — that covers the minimal liftable unit (ghmm/process.py+utils.py)that most consumers will vendor first.
tests/so the vectors are exercised against this repo's ownreference modules — otherwise the vectors themselves are unverified, and a wrong vector is worse
than a missing one.
Deriving the expected values from the reference implementations makes the vectors a regression
baseline rather than an independent check of the reference code. That is probably the right trade
(they exist to police consumers' forks), but worth doing knowingly, and worth spot-checking a few
values against hand computation.