The detector computes a minimum window wall-time per candidate plateau, from that plateau's own data, then uses it to pick which plateau becomes the reported steady state. So the certified metric value depends on a (self-referenced) data-derived threshold, not a published constant. It also diverges from the ratified fixed minimum in the policy 6.2
I think the statistical intent is sound but the mechanism may not be appropriate for the benchmarks. While it's understandable to extend/apply min duration to steady-state plateaus, if done, it'd make sense to use the fixed duration requirement, i.e 600s minimum per plateau.
The specific section of reference is in the min_steady_duration routine:
|
cov_b = cov(tpot_p50) if len(tpot_p50) >= 2 else 0.0 |
|
spans = [ |
|
(sp.last_issue_ns - sp.first_issue_ns) / 1e9 |
|
for sp in window |
|
if sp.last_issue_ns > sp.first_issue_ns >= 0 |
|
] |
|
tau_sp = median(spans) if spans else 0.0 |
|
lat = pooled(series, lo, hi, "latency_ns") |
|
l_p90 = percentile_lower(sorted(lat), 0.90) / 1e9 if lat else 0.0 |
|
|
|
kstar = max(math.ceil((CI_Z_95 * cov_b / MIN_DUR_EPS) ** 2), MIN_DUR_KSTAR_FLOOR) |
|
# The precision term binds only when noise demands more batches than the |
|
# trend floor. At k* == MIN_TREND_N, the trend requirement is already met. |
|
# Then relaxation and floor control duration. This keeps a clean minimal |
|
# plateau valid when it meets the wall-time floor. |
|
t_prec = kstar * tau_sp if kstar > MIN_DUR_KSTAR_FLOOR else 0.0 |
|
t_relax = MIN_DUR_RELAX_MULT * l_p90 |
|
min_s = max(t_prec, t_relax, MIN_DUR_FLOOR_S) |
Given the parameters up to the min time calculation are self-referenced (w.r.t the plateau itself), each plateau is judged against a bar derived from its own noise/latency; plateaus aren't compared on a common yardstick.
In the meantime, we can default enforce_min_duration=False
|
dest="enforce_min_duration", |
until any dynamic threshold is ratified in the policies.
@nv-alicheng @arekay-nv @mrmhodak @nvashutoshd @hans-intel
The detector computes a minimum window wall-time per candidate plateau, from that plateau's own data, then uses it to pick which plateau becomes the reported steady state. So the certified metric value depends on a (self-referenced) data-derived threshold, not a published constant. It also diverges from the ratified fixed minimum in the policy 6.2
I think the statistical intent is sound but the mechanism may not be appropriate for the benchmarks. While it's understandable to extend/apply min duration to steady-state plateaus, if done, it'd make sense to use the fixed duration requirement, i.e 600s minimum per plateau.
The specific section of reference is in the
min_steady_durationroutine:endpoints/src/inference_endpoint/metrics/steady_state_diagnostics.py
Lines 1633 to 1650 in f1100cf
Given the parameters up to the min time calculation are self-referenced (w.r.t the plateau itself), each plateau is judged against a bar derived from its own noise/latency; plateaus aren't compared on a common yardstick.
In the meantime, we can default
enforce_min_duration=Falseendpoints/src/inference_endpoint/metrics/steady_state_diagnostics.py
Line 2147 in f1100cf
@nv-alicheng @arekay-nv @mrmhodak @nvashutoshd @hans-intel