Skip to content

[engine] carry typed-action-contract schemas on the Sources payload - #70

Merged
cmttt merged 2 commits into
mainfrom
ls/typed-contract-schemas-etcd
Jun 24, 2026
Merged

[engine] carry typed-action-contract schemas on the Sources payload#70
cmttt merged 2 commits into
mainfrom
ls/typed-contract-schemas-etcd

Conversation

@cmttt

@cmttt cmttt commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Problem

#55 added the typed-action-contract graph specializer + a schema loader that reads schemas from a filesystem directory (OSPREY_SCHEMAS_DIR, else OSPREY_RULES_PATH/schemas). A worker whose rules come from etcd has no schemas directory on disk, so resolve_schemas_dir() returns None and the specializer never activates there — even with the prune/shadow allowlist set.

Approach

Carry the schema JSON (schemas/*.json + schemas/types/*.json) on the same Sources payload as the rules, so it rides the existing rules→etcd→worker rail, and have the loader read schemas from the in-memory Sources. The serialized payload (to_dict) is byte-identical to today when there are no schemas, so the publisher/provider/deployer are untouched.

Changes

  • Sources — a separate _schemas map (kept out of the .sml compile path): from_path (anchored glob('schemas/**/*.json')), to_dict (merge; no-op when empty), from_dict (partition schemas/ keys before the .sml-asserting add_source), hash() (fold schemas in so a schema-only edit invalidates dedup and the worker reloads), and schemas()/get_schema() accessors.
  • schema_loader — extract a source-agnostic parse_schema(raw, ref_reader, where); load_schema stays the disk wrapper (behavior unchanged); add load_schema_for_action_from_sources(action, schemas_map) with a key-space $ref traversal guard.
  • typed_contract_dispatchload_and_register_specialized_graphs(..., schemas=None) reads from the in-memory map when present, else falls back to the on-disk dir (local-dev preserved).
  • both engines — pass self._execution_graph.validated_sources.sources.schemas().

Rollout (worker-before-deployer, one-way)

The payload format gains schemas/ keys, which an old worker's from_dict would route into the .sml-asserting path. So the worker must run this code before any payload carrying schema keys is published. Until schemas are pushed, to_dict is byte-identical to today, so the change is fully back-compatible during the rollout window.

Tests

72 passing: Sources round-trip + back-compat (no-schema payloads, schema-key payloads, hash-on-schema-edit), schema_loader disk/Sources parity + $ref traversal guards, dispatch source-selection (Sources vs disk vs disabled). Existing disk-loader tests unchanged.

#55's schema loader reads schemas from a filesystem directory (OSPREY_SCHEMAS_DIR /
OSPREY_RULES_PATH/schemas). A worker sourced from etcd has no schemas dir on disk, so
the specializer never activates. Carry the schema JSON (schemas/*.json +
schemas/types/*.json) on the same Sources payload as the rules, so it rides the existing
sources rail, and read schemas from the in-memory Sources.

- Sources: a separate `_schemas` map (kept out of the .sml compile path) wired through
  from_path (glob schemas/**/*.json, anchored at root), to_dict (no-op/byte-identical when
  empty), from_dict (prefix-partition before the .sml-asserting add_source), and hash()
  (so a schema-only edit invalidates dedup and the worker reloads). schemas()/get_schema().
- schema_loader: extract a source-agnostic parse_schema(raw, ref_reader, where); load_schema
  stays the disk wrapper (behavior unchanged); add load_schema_for_action_from_sources that
  reads from the in-memory map with a key-space $ref traversal guard.
- typed_contract_dispatch: load_and_register_specialized_graphs takes an optional schemas
  map and reads from it when present, else falls back to the on-disk dir.
- both engines pass self._execution_graph.validated_sources.sources.schemas().

Rollout is worker-before-deployer: an old worker's from_dict would hit the .sml assert on a
schema-bearing payload. Until schemas are pushed, to_dict is byte-identical to today.

72 tests: Sources round-trip + back-compat, schema_loader disk/sources parity + traversal,
dispatch source-selection. Existing disk-loader tests unchanged.
@cmttt
cmttt requested a review from haileyok as a code owner June 24, 2026 03:24
The async rules sink (AsyncRulesRunner.classify_one) called async_execute() against
engine.execution_graph directly — always the full graph — so the typed-action-contract
dispatch added in #55 (resolve_dispatch + shadow, wired into engine.execute()) was never
reached at runtime on the async worker: specialized graphs were registered at init but
never served. The gevent sink already routes through engine.execute(); this aligns the
async sink.

classify_one now calls self._engine.execute(...), which serves the specialized (pruned)
graph / shadow-diffs allowlisted actions, and the full graph otherwise (identical to the
prior direct call for non-allowlisted actions). Removed the now-unused async_execute import.

Added a regression test pinning the sink to the dispatch-aware engine method (no test
covered this call path, which is why the gap shipped).

Verified end-to-end locally with coordinator-asyncio + worker-asyncio running: an
allowlisted action served its specialized graph (prune) / ran shadow, schemas sourced from
the in-memory Sources with no schemas dir on disk.
@cmttt

cmttt commented Jun 24, 2026

Copy link
Copy Markdown
Owner Author

Verified end-to-end locally with osprey-coordinator-asyncio + osprey-worker-asyncio running (schemas sourced from the in-memory etcd Sources, no OSPREY_SCHEMAS_DIR):

Loaded 2 specialized graphs from Sources (prune=['guild_invite_created'] shadow=['guild_invite_created','guild_member_kicked'])
TAC-PROOF: action=guild_invite_created dispatch=PRUNE(serve specialized graph) registered_specialized=True
TAC-PROOF: action=guild_member_kicked dispatch=SHADOW(serve full + diff) registered_specialized=True

The first live run surfaced a real gap: the async sink (AsyncRulesRunner.classify_one) bypassed engine.execute() (called async_execute() on the full graph directly), so the #55 dispatch was a runtime no-op on the async worker. Fixed in cd7ccb0 (+ regression test). After the fix, both actions hit the specialized path through the live coordinator→worker bidi transport.

@cmttt
cmttt merged commit 627d9e5 into main Jun 24, 2026
1 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant