[Bug] TOPK-BOUND-VIOL clamps live_n above baked_stride under sustained agent load (v0.5.6, GB10)

[NotSamClemens]

Summary

On a DGX Spark running Entrpi DS4 v0.5.6 (df641a7c4358dd6ca3b5acb46cf884a7d42066ed), sustained long-context agent workloads repeatedly trigger:

ds4: TOPK-BOUND-VIOL count=... live_n=... baked_stride=...

The server does not crash. It continues decoding after clamping the live scan bound to the baked stride. Because top-k selection is part of the model computation, affected outputs appear correctness-suspect even when the generated text remains fluent.

This looks related to the kernel family in #9, but it is a distinct observed failure mode: #9 reports an illegal-address crash on v0.5.4; this report concerns repeated non-crashing bound violations on v0.5.6 under real agent traffic.

Environment

• Hardware: NVIDIA DGX Spark, GB10 / SM121
• OS: Ubuntu 24.04.4 LTS, aarch64
• NVIDIA driver: 580.173.02
• DS4: v0.5.6, commit df641a7c4358dd6ca3b5acb46cf884a7d42066ed
• Base: DeepSeek-V4-Flash-0731 imatrix GGUF
• Speculation: matching 0731 DSpark drafter, --no-mtp
• Server context: 400000
• Binding: loopback, proxied through llama-swap

Observed behavior

The invariant violation appeared in two independent long-horizon workloads.

OpenCode coding workload

At roughly 39K context, the upstream DS4 log repeatedly reported values in this range:

live_n=9839..9843
baked_stride=8193..8195

The server continued decoding at roughly 18.4 tok/s instead of crashing.

Open WebUI multi-agent research workload

During concurrent sub-agent work, the violation appeared across more than one session/kernel shape:

live_n=8509  baked_stride=8508
live_n≈4651  baked_stride≈4635..4636

The global violation counter exceeded 32,000 during the run. Later phases continued without new violations, but that cannot validate tokens generated while the guard was firing.

Relevant source behavior

In ds4_cuda.cu, the running tag reads a live scan bound from the layer substrate and compares it with the captured row stride:

uint32_t n = ls ? ls->n_index_comp : n_comp;
if (ls != NULL && (n == 0u || n > n_comp)) {
    if (tid == 0u) {
        atomicAdd(&g_topk_bound_viol_count, 1u);
        g_topk_bound_viol_n = n;
        g_topk_bound_viol_stride = n_comp;
    }
    if (n == 0u) return;
    n = n_comp;
}

Source: https://github.com/Entrpi/ds4/blob/df641a7c4358dd6ca3b5acb46cf884a7d42066ed/ds4_cuda.cu#L15499-L15517

The polling path then emits TOPK-BOUND-VIOL after command synchronization:

https://github.com/Entrpi/ds4/blob/df641a7c4358dd6ca3b5acb46cf884a7d42066ed/ds4_cuda.cu#L3441-L3460

My reading is that n = n_comp prevents the suspected illegal access, but also means candidates beyond the baked stride are not included in that scan. Please correct me if that interpretation is wrong.

Expected behavior

• live_n should never exceed the stride assumed by the active captured kernel; or
• the kernel/capture should be rebuilt or selected for the new live bound before executing top-k; and
• a long-context/concurrent soak should complete with zero bound violations.

Correctness concern

The process remains operational, so normal health checks and HTTP success do not detect this condition. Fluent output and normal decode speed also do not establish correctness after the clamp fires.

This makes the current tripwire more than diagnostic noise: unless the clamp is mathematically equivalent to the intended selection, affected generations cannot be treated as faithful model output.

v0.5.6.2 status

I have not yet reproduced the workload on v0.5.6.2. Public source inspection shows that the relevant guard remains present; ds4_cuda.cu is identical between v0.5.6.1 and v0.5.6.2. The v0.5.6.2 changes are in the server/Codex integration path, not this selector.

Questions

1. Is live_n > n_comp expected under continuous-batch capture replay?
2. Is the n = n_comp fallback mathematically output-equivalent, or does it truncate the candidate set?
3. Is there an existing branch or planned fix that reselects/recaptures the kernel when the live bound grows?
4. Would a smaller deterministic reproducer or additional diagnostics be useful? I can retest on v0.5.6.2 and provide exact logs.