Repository navigation
Share deployment topology mapping between benchmarks and evals - #1304
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit eb61c96. Configure here.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Benchmark and eval ingestion had separate topology calculations. Eval could multiply TP by EP and count mirrored aggregate metadata as two workers, making a four-GPU SGLang eval appear as 32 GPUs.
Route performance results, individual eval artifacts, and collected eval results through one shared deployment-topology resolver. It owns physical GPU counting, worker multiplicity, aggregate role mirroring, and disaggregation. AgentX SGLang TP/EP share devices; physical node placement alone does not imply separate prefill/decode engines. Explicit GPU counts remain authoritative, and historical fixed-sequence count conventions are preserved.
This addresses the eval mappings found in SemiAnalysisAI/InferenceX#3701, SemiAnalysisAI/InferenceX#3782, and SemiAnalysisAI/InferenceX#3744. Existing database rows are not rewritten; already-ingested incorrect configurations require a separate audited reconciliation.
Note
Medium Risk
Changes ETL config identity (GPU counts, disagg flags, aggregate mirroring) for benchmarks and evals; incorrect mapping could mis-key configs until reconciliation, though behavior is heavily test-covered.
Overview
Introduces
resolveConfigTopology()inpackages/db/src/etl/topology.tsand routes benchmark, individual eval, and agg eval mappers through it, replacing duplicatedresolveParallelism/buildEvalConfiglogic.Config identity fixes: legacy role GPU fallbacks now multiply per-worker TP×EP (or AgentX TP×PP×PCP) by worker replicas on both paths; SGLang/Dynamo-SGLang AgentX treats EP/DCP as sharing TP devices; single-node SGLang rows with mirrored 1P/1D role fields collapse to one aggregate config (zero role workers, same GPU count on both sides). Disaggregation is driven by framework aliases and non-zero decode worker pools, not
is_multinodeor prefill workers alone.Docs in
docs/data-pipeline.mddescribe the shared resolver and note that fixes apply on future ingest only—existing rows need audited reconciliation, not blind re-ingest.Reviewed by Cursor Bugbot for commit eb61c96. Bugbot is set up for automated code reviews on this repo. Configure here.