[engine] carry typed-action-contract schemas on the Sources payload - #70
Merged
Conversation
#55's schema loader reads schemas from a filesystem directory (OSPREY_SCHEMAS_DIR / OSPREY_RULES_PATH/schemas). A worker sourced from etcd has no schemas dir on disk, so the specializer never activates. Carry the schema JSON (schemas/*.json + schemas/types/*.json) on the same Sources payload as the rules, so it rides the existing sources rail, and read schemas from the in-memory Sources. - Sources: a separate `_schemas` map (kept out of the .sml compile path) wired through from_path (glob schemas/**/*.json, anchored at root), to_dict (no-op/byte-identical when empty), from_dict (prefix-partition before the .sml-asserting add_source), and hash() (so a schema-only edit invalidates dedup and the worker reloads). schemas()/get_schema(). - schema_loader: extract a source-agnostic parse_schema(raw, ref_reader, where); load_schema stays the disk wrapper (behavior unchanged); add load_schema_for_action_from_sources that reads from the in-memory map with a key-space $ref traversal guard. - typed_contract_dispatch: load_and_register_specialized_graphs takes an optional schemas map and reads from it when present, else falls back to the on-disk dir. - both engines pass self._execution_graph.validated_sources.sources.schemas(). Rollout is worker-before-deployer: an old worker's from_dict would hit the .sml assert on a schema-bearing payload. Until schemas are pushed, to_dict is byte-identical to today. 72 tests: Sources round-trip + back-compat, schema_loader disk/sources parity + traversal, dispatch source-selection. Existing disk-loader tests unchanged.
The async rules sink (AsyncRulesRunner.classify_one) called async_execute() against engine.execution_graph directly — always the full graph — so the typed-action-contract dispatch added in #55 (resolve_dispatch + shadow, wired into engine.execute()) was never reached at runtime on the async worker: specialized graphs were registered at init but never served. The gevent sink already routes through engine.execute(); this aligns the async sink. classify_one now calls self._engine.execute(...), which serves the specialized (pruned) graph / shadow-diffs allowlisted actions, and the full graph otherwise (identical to the prior direct call for non-allowlisted actions). Removed the now-unused async_execute import. Added a regression test pinning the sink to the dispatch-aware engine method (no test covered this call path, which is why the gap shipped). Verified end-to-end locally with coordinator-asyncio + worker-asyncio running: an allowlisted action served its specialized graph (prune) / ran shadow, schemas sourced from the in-memory Sources with no schemas dir on disk.
Owner
Author
|
Verified end-to-end locally with The first live run surfaced a real gap: the async sink ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#55 added the typed-action-contract graph specializer + a schema loader that reads schemas from a filesystem directory (
OSPREY_SCHEMAS_DIR, elseOSPREY_RULES_PATH/schemas). A worker whose rules come from etcd has no schemas directory on disk, soresolve_schemas_dir()returnsNoneand the specializer never activates there — even with the prune/shadow allowlist set.Approach
Carry the schema JSON (
schemas/*.json+schemas/types/*.json) on the sameSourcespayload as the rules, so it rides the existing rules→etcd→worker rail, and have the loader read schemas from the in-memorySources. The serialized payload (to_dict) is byte-identical to today when there are no schemas, so the publisher/provider/deployer are untouched.Changes
Sources— a separate_schemasmap (kept out of the.smlcompile path):from_path(anchoredglob('schemas/**/*.json')),to_dict(merge; no-op when empty),from_dict(partitionschemas/keys before the.sml-assertingadd_source),hash()(fold schemas in so a schema-only edit invalidates dedup and the worker reloads), andschemas()/get_schema()accessors.schema_loader— extract a source-agnosticparse_schema(raw, ref_reader, where);load_schemastays the disk wrapper (behavior unchanged); addload_schema_for_action_from_sources(action, schemas_map)with a key-space$reftraversal guard.typed_contract_dispatch—load_and_register_specialized_graphs(..., schemas=None)reads from the in-memory map when present, else falls back to the on-disk dir (local-dev preserved).self._execution_graph.validated_sources.sources.schemas().Rollout (worker-before-deployer, one-way)
The payload format gains
schemas/keys, which an old worker'sfrom_dictwould route into the.sml-asserting path. So the worker must run this code before any payload carrying schema keys is published. Until schemas are pushed,to_dictis byte-identical to today, so the change is fully back-compatible during the rollout window.Tests
72 passing:
Sourcesround-trip + back-compat (no-schema payloads, schema-key payloads, hash-on-schema-edit),schema_loaderdisk/Sources parity +$reftraversal guards, dispatch source-selection (Sources vs disk vs disabled). Existing disk-loader tests unchanged.