[binaryaaron]

this is comically large and the result of a few attempts to do a heavy, semi-autonomous agent run. i will be making changes to remove larges swaths of code and weird tests from the repo, bc the initial +130k loc is absurd on its face.

Summary

This PR moves Anonymizer's production pipelines (Rewrite, detection, Substitute) from hand-written per-route controllers onto a typed Graph SDK, behind the unchanged Anonymizer API: the public DataFrame behavior is preserved under REQ 1 except the accepted deltas listed in the status block. DataDesigner execution stays behind NddAdapter.run_workflow().

anonymizer.graph is not a compatibility contract yet. Until the Graph SDK is published, its names, __all__ order, signatures, defaults, exception constructors, class pickles and import cost may change in any commit. Tests that pin them are refactor safety nets and are re-recorded with the change. The contract is the Anonymizer API and config of 0.4.0.

This branch (aagonzales/graph-sdk/feat/main) is the long-lived base of the Graph SDK feature line, not a merge candidate for now. Later work, including PR2, builds on it and lands here as PRs.

Done: Rewrite, detection and Substitute run graph-owned end to end, with their legacy controllers deleted; a scoped logging correction keeps record content out of library diagnostics. The history is squashed into 15 reviewable commits; #314–#316, a main merge and #318 followed. Evaluation, native admission and Session are PR2+.

For dogfooders and reviewers

This branch moves the production pipelines onto the Graph SDK while preserving the public DataFrame
API. Its history was rewritten once, on 2026-10-20, into 15 commits with the same tree as 2fc6d810; otherwise it
only fast-forwards. Test at the latest commit and state the SHA you ran in every comment (current pin: e0d8b317).
Setup (in this order, details in tools/dogfood/README.md):
uv sync --frozen --extra notebooks; start the detector server
(uv run python tools/serve_gliner.py --host 127.0.0.1 --port 8001 — no bearer token is needed unless you set
ANONYMIZER_LOCAL_GLINER2_TOKEN when starting it); export the provider key named in
tools/dogfood/providers.yaml (NVIDIA_INFERENCE_API_KEY; or edit the file for your provider). Smoke test:
pytest -m compat (~1 min), then python examples/graph_sdk/local_workflows.py (provider-free).
Dogfood kit: tools/dogfood/dogfood_run.py runs one route in one environment and writes a comparable
result bundle; tools/dogfood/compare.py diffs two bundles. Run the same route against the released wheel in
a second environment at matching versions (pandas==2.3.3, data-designer==0.9.4) and diff; against a
pandas 3 install the only expected differences are the trace-column dtype rows listed below.
How to approach it, in this order. No line-by-line review yet; the branch is being cut down substantially.

1. Dogfood with any data, at the latest commit. Point your agent at this section and just verify it works
   on your end: detection+Redact, Substitute, Rewrite via run and preview, diffed against the released 0.4.0
   with tools/dogfood/. Highest-value surfaces: Substitute and Rewrite (replacement maps), custom
   detectors/runners, nullable or non-string columns, Ctrl-C mid-run. A one-line "clean at , pool "
   comment is as useful as a diff.
2. Latency and quality benchmarks from the team's experiments repo at the same pin and model pool as your
   release baseline; post the deltas (instructions in the team channel).
3. Build something small with the public SDK — for example, how you would adapt work you are doing now, or
   a case like integrating with a partner project — and tell us where the semantics or composability surprised
   you, where docs/concepts/graph-sdk.md and the behaviour disagree, and what you could not express. Tag
   [sdk-semantics]; those are not parked.
   What to look at: result.failed_records; trace_dataframe["_replacement_application"] — a non-zero
   skipped_span_count means original text survived; a null *_rewritten with needs_human_review=True is a
   rewrite WITHHELD by design, not a crash; output column dtypes vs the release; leakage_mass and
   needs_human_review on Rewrite; the console log at DEBUG for any entity value. Always state the model pool
   and provider you used — results differ by model. Rewrite with nemotron-3-super fails structured outputs on
   both the branch and 0.4.0; use the default pool for Rewrite comparisons.

What matters now is correctness and discrepancies with main / the released package: wrong outputs,
crashes, dtype/NaN/row-handling defects, PII or exception content leaking into logs/traces. Architecture,
naming, duplication, size and style comments are not useful yet — an active minimization effort owns them
and they will be parked (not ignored) — see the triage replies in the footer.

File findings as comments on this PR — one finding per comment, tagged [breaks-compat],
[wrong-output], [crash], [privacy], [docs-gap], or [sdk-semantics] (phase 3), with: the exact SHA you reviewed
against, the topic area (route/surface), smallest repro (provider-free if possible), and expected vs actual
(one line each).
All coordination for this branch happens on this PR; there is no separate guide or issue form.

Status, known-open, and accepted behavior deltas

Branch role (maintainer, 2026-10-21): this branch is the long-lived base of the graph-SDK feature line. It is not
scheduled to merge into main; later work, including PR2, lands here as reviewed PRs, and main is merged in as its
PRs land. Dogfood it as a moving base: the pin in the dogfooder block is the current head, and the known-open and
accepted-delta lists below describe that head.

Live validation (2026-10-19, operator dogfood vs released 0.4.0 at 5473e8ef; first run 2026-10-16 at
0842f5c5 reported the same outcome): detection+Redact, Substitute and Rewrite run end to end on two live
provider pools via run and preview; outputs match the release on columns, dtypes (under aligned
dependencies; pandas 3 differences are the accepted deltas below), entity coverage, replacement-text leakage
and log hygiene. Timing within 0.7–1.5× of the release across routes; the Substitute replacement-map
serialization fixed at e309c617 holds (10 requests in flight). Evidence retained in the campaign's records.

Size (maintainer decision, 2026-10-21): the original targets were +10K, then +5K, non-test Python lines over the
0.4.0 baseline 30b6d82a. Measured at pin 188ef369: +32,760 non-test (+31,642 excluding tools/) and
+57,383 test. The targets were revised rather than met because the branch was reduced in stages on top of the
reviewed implementation; a rebuild was ruled out. Continuing reduction and simplification will be done in commits
directly to this branch or in PRs made to it directly (the first three, caller-thread execution for every synchronous
body, an eager anonymizer.graph facade and a DataDesigner version range, have landed), on branches named
<author>/graph-sdk-<name>. Size, naming and layering comments stay parked until that work lands (see triage replies).

Known-open — don't file these:

• Evaluation cutover, native host execution/admission, and Session/identity/matching are
  deferred to PR2+ (maintainer scope ruling, 2026-10-16). evaluate() ships on its existing
  legacy path — behavior-preserving; don't file "evaluation isn't graph-owned" findings.
• PR1's remaining work is simplification: the implementation is being substantially reduced and
  restructured. Internal modules, names, and layering will churn heavily — architecture/naming
  feedback stays parked (see triage replies).
• Detection batch speed matches the release with real endpoints (1.00–1.02× at 100–200 rows; the earlier
  internal ~2.1× measurement came from a quadratic scan in the final-output assembly, fixed in 0cc24a4a).
  About 7–8 ms/row of CPU orchestration remains on top of model time. Don't file timing findings under
  1.1× of the release; do file anything above it with row count and model pool.
• The anonymizer.graph SDK surface is not a contract in this PR: its names, export order, signatures,
  defaults, exception constructors, pickles and import cost may change until the SDK is published under the
  RFC's publication gate (REQ 13). Tests that pin them are refactor safety nets, re-recorded with the change
  that moves them. Continuity, adaptive retrieval, decision waits, state revisions and absence queries have
  moved to PR2+. A changed or removed graph export is not a finding; the shipped Anonymizer/config API and
  the documented result columns are the stable target.
• Running two Anonymizer processes at once prints OSError: [Errno 98] Address already in use from
  DataDesigner's metrics server, then "OpenTelemetry metrics are unavailable; continuing without them."
  Harmless.
• Substitute can leave an entity's original text in place when the replacement generator omits it from
  its map; the trace's skipped_span_count shows it. Pre-existing in 0.4.0, model-dependent (seen with
  nemotron-3-super, not with the default pool). Tracked for PR2; still report it with your model pool so
  we see the frequency.

Accepted behavior deltas (a released-vs-branch diff covered by a row here is not a finding):

Delta	Routes affected	Accepting checkpoint SHA
Rewrite trace: internal _needs_repair / _rewrite_replacement_ready columns are bool (release: object, NaN on no-entity rows); on the staged path _needs_repair is False for withheld rows (release True), since the withheld text is not repaired	Rewrite, trace_dataframe only	8f5a9b74, d34ccdfb
Redact: an empty or statically empty format_template (e.g. "" or "{label:.0}") raises a configuration error, and a label template that renders empty during replacement raises instead of silently keeping the original entity text (0.4.0 accepted these and could leave the entity in place). Rationale: fail-closed — an empty template would publish the original PII. Migration: supply a non-empty template (default [REDACTED_{label}])	Replace (Redact)	0c1b719a
Rewrite trace: QA columns (_privacy_qa_reanswer, _quality_qa_reanswer, _quality_qa_compare) hold empty structures instead of NaN on rows with no entities; a batch where no row has entities adds these 4 trace columns (release: absent)	Rewrite, trace_dataframe only (both schedules, run/preview, evaluate)	b2c9c7bf
pandas 3 only: evaluate() on a Rewrite preview returns the trace _anonymizer_record_id column as str dtype instead of object	evaluate, trace_dataframe only	b2c9c7bf
A caller-supplied detector that returns duplicated rows, with a preview num_records smaller than the returned entity rows: the branch returns the requested n rows and drops the extra row (failed_records empty); 0.4.0 and main crash with a raw pandas ValueError in _join_new_columns	Rewrite route with caller-supplied detectors, preview (Redact/Substitute preview behave the same on both)	b2c9c7bf
evaluate() on a Substitute/Redact result whose caller-supplied detector duplicated rows returns one row per result row (7 in the duplicated-detector case); 0.4.0 and main return 9 because their replace-judge merge multiplies the duplicated rows	evaluate after Replace with caller-supplied detectors	3c7c7ccc
Rewrite, evaluate() and result.rewrite_config use the privacy goal as admitted (0.4.0 used the goal object as mutated by the detection stage)	Rewrite, evaluate	a67ef6bd
A standalone combined-finalizer row whose repair decision is missing, null or non-boolean fails that record (EffectRejected, surfaced as a failed record) instead of 0.4.0's silent projection of the first evaluation state; unreachable through built-in routes, only via caller run_workflow overrides or hand-built rows	Rewrite combined schedule with caller run_workflow overrides	1301c619
Context variables set inside caller-supplied code (detectors, injected runners, build_graph, adapter run_workflow overrides) are not visible to the caller after run/preview/evaluate return; values set before the call are visible inside (0.4.0 ran caller code in the caller's live context, so writes leaked out). Rationale: every public entry runs in a copy of the caller's context, copy-in and no copy-out, the only semantics an async entry can also offer. Migration: pass results back through return values or your own objects, not through context variables	All routes with caller-supplied code	2fc6d810

What has landed

The 15 commits (history squashed 2026-10-20; tree unchanged)

Commit	What it adds
a75b1dee	Graph values, workflows and static composition; the anonymizer.graph facade
404f64fa	Activation, data graphs and bounded plan preparation
6c14a4fa	Typed requests, context binding and observations
729a577c	Initial bindings, blocking providers and foreign transport
849a203a	Execution of admitted graphs under a bounded runtime
d684c273	Evidence qualification against execution accounting
bceffa9d	Inspection and extensions; public SDK guide
3de54364	DataDesigner bridge to graph requests
e624f10d	Public facade compatibility tests, before the route cutover
0b51cfef	Detection and Redact under graph ownership (route cutover, part 1)
ced49332	Rewrite under graph control; facade routing (part 2)
1fdb2937	Substitute generation under graph control (part 3; suite green from the next commit)
372bbd0b	Logging: keep sensitive payloads out of library diagnostics
60959fa7	Cutover inventory and preview-branch rules
4bded5e6	Branch-vs-release dogfood kit

Current numbers

Measure	Value
Python growth vs baseline 30b6d82a	+32,760 non-test / +57,383 test (net, *.py, at 188ef369)
Full suite	5,661 passed at 3a208200 (pandas 2 and 3); parity corpus vs 0.4.0: 0 blocking
CI	Run 39620094519 at 188ef369
Wheel	at 2fc6d810: one identical wheel on Python 3.11–3.13; compat 265 from the installed wheel
Live dogfood vs 0.4.0	see Live validation in the status block
Substitute benchmark	zero-latency mock (1,025 rows): median 0.909×; 250 ms-latency mock, 16 rows: map stage 0.23 s (0.4.0: 0.23 s)

Remaining work

1. PRs into this branch (branch <author>/graph-sdk/<type>/<name>, the naming order proposed in docs: allow topic prefixes in branch names for long-lived feature lines #312). Landed: refactor(graph): make anonymizer.graph an eager facade #314 (eager anonymizer.graph facade), build: accept data-designer >=0.9.4,<0.10 with capability-checked hooks #315 (DataDesigner >=0.9.4,<0.10), fix(graph): run every Anonymizer body on the caller's thread #316 (caller code and builtin DataDesigner waits on the caller's thread), test(graph): accept CPython patch-release differences in graph SDK tests #318 (tests pass on every supported Python 3.13 patch release).
2. Main: merged in by merge commit (first at 188ef369). Later main PRs come in the same way as they land, with a port PR when they touch graph-owned code.
3. Size: see the size paragraph in the status block.
4. Deferred to PR2: async entry point, a concurrency statement, sharding primitives; residual logging conformance (dependency loggers, exporters, telemetry).

Verification

Each PR into this branch gets the full suite (pandas 2 and 3), repository checks and independent review.

Related Issue and Plan

Maintainer-directed Graph SDK implementation; no linked issue required. See the public SDK guide, cutover inventory and provider-free example.

Type of Change

• New feature
• Refactoring
• Documentation update

Contributor Checklist

• Conventional Commit title and maintainer-owned no-issue reason
• Public guide, example and cutover inventory linked
• Final reviews before un-draft (public API and bundled skill impact, fixture/artifact privacy, conformance and identical-wheel qualification, minimization closeout): see Remaining work

Triage replies (paste verbatim)

Parked:

Parked, not ignored: this is architecture/naming/size/style feedback, which the minimization
effort owns right now. It goes on the parked list and will be revisited after minimization
completes. Thanks for flagging it.

Needs repro:

Needs a repro before it can enter the queue: please add the exact SHA you ran and the smallest
script/config/data that reproduces it (provider-free if possible), with expected vs actual, one
line each. Re-label or comment once it's attached and we'll triage.
SDK semantics:

Logged as [sdk-semantics] input to the public-surface decision; not a PR1 blocker and not parked.
If it is also a wrong output or a crash, re-tag it so it enters the correctness queue.

Model-dependent:

Thanks — reproduced as model-dependent: it occurs with and not with the default pool in our
runs. Logged on the PR2 backlog with your model config attached; not a PR1 blocker. If you see it with
the default pool, please re-flag.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

This PR was closed automatically when its branch was renamed to aagonzales/graph-sdk/feat/main. The same branch, history and description continue as #319. Please read, dogfood and comment there.