You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Map missing agentic replay and video quality eval fields reported by Shield drift check - #159
Maps keys that Shield metric drift alerts reported as unmapped:
Agentic replay (Kimi, GLM, Llama): 12 new benchmark metrics (*_e2e_norm_intvty, p95_isl/p95_osl, p99_adj_*, prefix_cache_*_tokens_measured) and 2 new config fields (repetition, trace_idle_gap_cap_seconds)
MiniMax-H3 video quality eval (51 keys): summary.* counts and elapsed_seconds as eval metrics. Settings, thresholds and per-sample detailed_results as config.
Speculative decoding sweep (Kimi-K2.7-Code, Qwen3-VL-32B, gemma-4-31B): route kind: aiperf_spec_decode sections as benchmarks. Add acceptance_rate, mean_accepted_length and output_throughput as metrics and output_len as config. Use public_dataset as the dataset name.
run_type change:agentic_traces and aiperf_spec_decode rows now get their own run_type (same as the section kind) instead of benchmark. They reuse LLM metric names such as mean_ttft_ms for a different workload, so they no longer mix with token sweeps in consumers that filter on run_type = 'benchmark'. Token sweeps (vllm, benchmarks) stay benchmark, evals stay eval, measurements keep step_name = 'benchmark', and rows keep config_params.report_section_kind.
Agentic rows already stored (80 rows in 51 runs since 2026-08-31, all stamped report_section_kind = 'agentic_traces') stay under benchmark, because sw_test is insert-only. After this merges, a tt-shield backfill re-collects those runs (plus past spec-decode runs) under the new run_type values. Old copies can be excluded from benchmark queries with report_section_kind IS DISTINCT FROM 'agentic_traces'.
evaluation_type is a run setting in the new video-quality payload (video_quality_eval.json:14), but it is absent from both the eval metric list and _CONFIG_FIELDS_BY_STEP["eval"]. Because these declarations are the drift-checker whitelist, this field remains unmapped and the reported drift is not fully resolved; add it as eval configuration and cover it in the regression assertion.
… drift check
- Route kind:aiperf_spec_decode sections (Kimi-K2.7-Code, Qwen3-VL-32B,
gemma-4-31B) as benchmarks; add acceptance_rate, mean_accepted_length
and output_throughput metrics, output_len as config, and use
public_dataset as the dataset name.
- Add tput_input, tput_input_ratio and tput_input_check target checks
(gemma-4-12B/31B, Qwen3.8-27B, MiniMax-M3).
Both sections reuse LLM metric names (mean_ttft_ms, ...) for a different
workload, so under run_type 'benchmark' they mix with token sweeps in
consumers that filter on it. Token sweeps (vllm, benchmarks) stay
'benchmark' and evals stay 'eval'; measurements keep step_name
'benchmark' and rows keep config_params.report_section_kind.
_SECTION_RUN_TYPES now lists vllm, benchmarks and evals too, and both
_process_benchmarks and _process_evals read it, instead of a silent
'benchmark' default plus a hardcoded 'eval'. A test requires its keys to
match _SECTION_METRIC_STEPS, so a newly routed kind cannot fall into
'benchmark' unnoticed. Legacy top-level lists have no section kind and
keep 'benchmark' / 'eval'. Stored rows are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Maps keys that Shield metric drift alerts reported as unmapped:
*_e2e_norm_intvty,p95_isl/p95_osl,p99_adj_*,prefix_cache_*_tokens_measured) and 2 new config fields (repetition,trace_idle_gap_cap_seconds)summary.*counts andelapsed_secondsas eval metrics. Settings, thresholds and per-sampledetailed_resultsas config.kind: aiperf_spec_decodesections as benchmarks. Addacceptance_rate,mean_accepted_lengthandoutput_throughputas metrics andoutput_lenas config. Usepublic_datasetas the dataset name.tput_input,tput_input_ratio,tput_input_checkrun_type change:
agentic_tracesandaiperf_spec_decoderows now get their ownrun_type(same as the section kind) instead ofbenchmark. They reuse LLM metric names such asmean_ttft_msfor a different workload, so they no longer mix with token sweeps in consumers that filter onrun_type = 'benchmark'. Token sweeps (vllm,benchmarks) staybenchmark, evals stayeval, measurements keepstep_name = 'benchmark', and rows keepconfig_params.report_section_kind.Agentic rows already stored (80 rows in 51 runs since 2026-08-31, all stamped
report_section_kind = 'agentic_traces') stay underbenchmark, because sw_test is insert-only. After this merges, a tt-shield backfill re-collects those runs (plus past spec-decode runs) under the newrun_typevalues. Old copies can be excluded frombenchmarkqueries withreport_section_kind IS DISTINCT FROM 'agentic_traces'.Adds regression tests.