Repository navigation
fix(agentx): role-aware, window-scoped ATOM cache rates with validity / 修正 ATOM 缓存命中率口径并标注有效性 - #3964
chunfangamd wants to merge 1 commit into
Conversation
… / 修正 ATOM 缓存命中率口径并标注有效性 The ATOM agentic result adapter summed prefix_cache_full_tokens over prefill and decode endpoints, reported lmcache_loaded_tokens (transfer volume) as CPU reuse, and set overall equal to GPU. The AIPerf metrics section also spans warmup. - Take endpoint roles from the launcher inputs (SRT_PREFILL/DECODE/AGG_ENDPOINTS). - Count prefill (or unified) endpoints over timeslices inside the profiling window. - CPU = prefix_cache_offload_tokens / full; external aliases CPU; overall = GPU + CPU. - Emit server_metrics.cache_validity so consumers know whether the rates are a complete, same-scope actual observation; refuse invalid or unprovable inputs. Refs: run 36734492117, #3543, #3158
andyluo7
left a comment
andyluo7
left a comment
There was a problem hiding this comment.
Requesting changes for one accounting issue. The endpoint-role accounting, profiling-window slicing, offload-based CPU reuse, and explicit refusal states are otherwise coherent. However, error accounting currently mixes warmup and profiling, so a harmless warmup failure can suppress an otherwise valid pricing observation.
I reviewed head 2db01926476a2dd93640c76e089787287575b45b; exact-head Lint and Tests are green. Deployment should remain blocked until phase-scoped error accounting is fixed and the corrected app-side guard is live.
中文
因一个计数问题请求修改。除此之外，endpoint role 计数、profiling 窗口切分、基于 offload 的 CPU 复用，以及明确的拒绝状态整体上是合理且一致的。但当前错误计数混合了 warmup 和 profiling，因此一次无害的 warmup 失败也会屏蔽原本有效的定价观测。
我审核的是提交 2db01926476a2dd93640c76e089787287575b45b；该提交的 Lint 和 Tests 均已通过。在修复按阶段统计错误的问题、并上线已修正的 app 侧保护逻辑之前，部署应继续保持阻塞。
| return { | ||
| "profiling_requests": len(records), | ||
| "profiling_prompt_tokens": sum(prompt) if len(prompt) == len(records) else None, | ||
| "error_records": request_accounting.get("records_error_dropped"), |
There was a problem hiding this comment.
records_error_dropped is not profiling-scoped: artifacts.py increments it before phase filtering, and atom.py rejects pricing whenever it is nonzero. A warmup-only error can therefore invalidate a profiling window whose admissions and prompt-token counts conserve exactly. Please track profiling errors separately (or increment after phase classification) and add a regression where warmup errors coexist with a fully valid profiling window.
中文
records_error_dropped 并不只统计 profiling 阶段：artifacts.py 在按阶段过滤之前就会递增它，而 atom.py 只要看到该值非零就会拒绝定价。因此，仅发生在 warmup 阶段的错误，也可能让一个 admission 数和 prompt token 数完全守恒的 profiling 窗口失效。请单独统计 profiling 错误（或在阶段分类后再递增），并添加一个回归测试，覆盖 warmup 有错误但 profiling 窗口完全有效的情况。
This branch has not been deployed
2 participants
Summary
ATOM prefill/decode (P/D) agentic results under-reported prefix-cache hit rates by about half. The ATOM result adapter (
inferencex-e2e/infx/results/agentic/backends/atom.py) had four problems:prefix_cache_full_tokensover prefill and decode endpoints. The decode worker re-counts each prompt on first admission without serving any reuse, so the denominator doubled.lmcache_loaded_tokens(LMCache transfer volume) as CPU reuse. ATOM records CPU reuse inprefix_cache_offload_tokens, which is disjoint from the HBMprefix_cache_cached_tokens.overallequal to GPU instead of GPU + CPU.stats.total, which for this export also spans the warmup phase.Found on run 36734492117 (see #3543; original recipe #3158).
Changes
SRT_PREFILL_ENDPOINTS/SRT_DECODE_ENDPOINTS, orSRT_AGG_ENDPOINTS). The mapping mirrorsinfx.bench.agentic.replay._server_metrics_urls. Only prefill (or unified) endpoints are counted. Without roles, the unified fallback applies only to a declared non-disagg single endpoint.summary.phase_time_ranges.profiling. There is nostats.totalfallback for priced rates. A slice straddling the window edge must be proven warmup-only bywarmup_metrics; sampling gaps, invalid slices and duplicate aliases are refused.externalis an alias of CPU (ATOM LocalCPUBackend).lmcache_loaded_tokensstays diagnostic only.full > 0,0 <= cached <= full,offload >= 0,cached + offload <= full(all finite). There is no clamping.server_metrics.cache_validity(nullfor non-ATOM adapters). It carriesstatus,pricing_eligible, observed/requested scope, CPU component, roles source, window and request-scope evidence, and reasons.KV_OFFLOADING=none.usage_prompt_tokens, no errors or cancellations); otherwise the rates are telemetry only (REQUEST_SCOPE_UNRESOLVED).tokens.prompt_totalkeeps its meaning (it still sums P and D prompt counters). Non-cache fields and other adapters are unchanged.test_process_agentic_result.pyalso carries formatter-only (ruff format) changes besides the schema-key update.Validation (head
2db01926476a2dd93640c76e089787287575b45b)mainreproduces every published cache field. The patch matches an independent window oracle:VALID_COMPLETEVALID_COMPLETEWINDOW_UNRESOLVEDREQUEST_SCOPE_UNRESOLVEDREQUEST_SCOPE_UNRESOLVEDrun.py→ producer CLI test proving that role env reaches the producer.test_eval_command.pyexcluded): 97 failed / 363 passed, vs base 97 failed / 322 passed. The failure IDs are identical and all are pre-existing environment failures (sandbox paths and modules outside this checkout). This is not a full green run; CI is the reference.infx/results/agentic.Not in this PR
Release dependency
Companion app PR: SemiAnalysisAI/InferenceX-app#1342. The two can be reviewed in parallel. Deploy order: the app validity guard, or an equivalent quarantine, must be live before results from this producer are priced. The current app prices a
nullcache rate as zero reuse.AI model disclosure
Claude Opus 5.5 (Anthropic), running as a Cursor IDE agent, investigated the issue, wrote the patch and tests, and drafted this description. Chun Fang reviewed the change through several review gates, and an independent review checked the patch and its counterexamples.
中文
修正 ATOM P/D agentic 结果中被低报约一半的前缀缓存命中率：
prefix_cache_offload_tokens，overall = GPU + CPU；server_metrics.cache_validity，标明这组值能否作为完整的实际计价输入；不完整或范围不明的数据只作遥测。用 run 36734492117 的五个点做了离线验证。本地相关套件仍有 97 项既有环境失败，不是全部通过。不包含历史回填，也没有修改计价默认值。app 防护（或等价隔离）必须先于新 producer 数据进入计价。