Repository navigation
Conversation
…ntine Add Pydantic models for fixed-sequence, AgentX, eval, and run-stats rows under infx/results/schema, with strictly typed identity fields and metric families validated by key pattern. Producers stamp result_schema_version 1 from a stdlib-only package. Collectors write invalid rows to rejected_rows.json, annotate them, and exit non-zero after writing the valid aggregate; collect workflows upload both files. Commit generated JSON Schemas under schemas/ with a drift test.
…ions Collectors validate rows without result_schema_version as version 1 and publish them stamped, so older checkouts keep publishing. Rows with any other version are quarantined as unsupported_version.
This was referenced Oct 7, 2026
Draft
Draft
Python serialized non-finite floats in a rejected row as bare NaN or Infinity tokens, so strict JSON parsers such as the InferenceX-app ingest could not read rejected_rows.json. Write those values as strings such as "NaN" and keep the rest of each rejected row unchanged.
Drop the never-overridden path parameter of quarantine.report, rename the VERSION key constant to VERSION_FIELD, and index the source and task keys that build_row always sets. Remove two assertions that repeated the annotation coverage of the benchmark collector test, and condense the result row contract section in both language versions of the guide.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Producer-side contract for the result rows InferenceX publishes. The app ETL (TypeScript) is unchanged here, except for one follow-up listed below.
infx/results/schema/): strict pydantic models for fixed-sequence, AgentX and eval rows and for run stats. They encode what producers emit today. Identity fields are typed strictly, and extra metric keys must match the latency/interactivity naming pattern or a known power key.result_schema_version: 1. A collector that receives an unstamped row from an older checkout validates it as v1 and stamps it. A row with an unknown version is rejected.collect_results,collect_eval_resultsandcalc_success_ratewrite the aggregate of valid rows. Invalid rows go torejected_rows.jsonwith the validation errors, with one::error::per row and exit code 1.rejected_rows_*files are uploaded as artifacts.python -m infx.results.schema exportwrites JSON Schema, committed underinferencex-e2e/schemas/. A test keeps the committed files equal to the export.infx.benchstays stdlib-only; no validation runs inside containers.docs/results-and-ingestion.mdhas a "Result row contract" section.Verification
isl, stringconcand NaN rows are quarantined while valid rows aggregate. Other tests cover the legacy stamping path, unknown versions, the eval path, and that the committed schemas match the export.App side
result_schema_versionout of metrics.rejected_rows_*and rows with an unknown version. Runs from current main ingest as before.rejected_rows.jsonwrites NaN/Infinity as strings so JavaScript can parse it.