[TRTLLM-17105][fix] Report a bounce failure only after the receiver has answered every sent chunk

Only the receiver's scatter of a received DATA writes the destination,
and the receiver answers (ACK or NACK) after that scatter's last write.
A failed request used to resolve kFAILURE at once, even while chunks
whose DATA was already sent were queued or running on the receiver, so
a caller that reused the destination could have it overwritten.

Contract: BounceResult gains `quiesced`. A failed request whose sent
chunks are not all answered now waits for those answers: it posts,
gathers and consumes credits no more, releases its staging and sends
its cancel as before, and resolves FAILURE with quiesced=true once the
last answer arrives. Answers match by (peer, rid, chunk, region), and
failRequest is idempotent, so a later forgetPeer keeps the first reason
and the answer deadline. The receiver drains the exec stream before
NACKing a failed launch, so a NACK never precedes a write.

In-doubt: with a sent chunk still unanswered, a receiver silent for
requestTimeoutMs after the failure or its last answer (the dead-peer
assumption the lease and quarantine already make), a kNoProgressTimeout,
or shutdown resolves the request with quiesced=false and a WARNING.

BounceTransferStatus moves to a header-only file; isCompleted() is true
on SUCCESS and on a quiesced FAILURE, and the status string marks an
in-doubt one.

The bounce reactor gtests get a 300 s per-test ctest TIMEOUT, so a hang
fails fast under its own name instead of consuming the 2700 s stage
budget.

The success path is unchanged: no new messages, locks, clock reads or
allocations. No wire or Python change.

Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com>

Author: chuangz0

cpp/include/tensorrt_llm/executor/transferAgent.h

@@ -374,9 +374,11 @@ class TransferStatus
 {
 public:
     virtual ~TransferStatus() = default;
-    /// NIXL and bounce statuses return true only on success; backends that cannot distinguish
-    /// failure from completion (mooncake) return true on any terminal state — use
-    /// wait()/getLastStatus for the outcome.
+    /// NIXL statuses return true only on success. Bounce statuses return true on success and on a
+    /// failure proven never to write the destination again (a source read by an orphaned local
+    /// gather may still be in flight); a failure without that proof stays false. Backends that
+    /// cannot distinguish failure from completion (mooncake) return true on any terminal state —
+    /// use wait() for the outcome.
     [[nodiscard]] virtual bool isCompleted() const = 0;
     virtual TransferState wait(int64_t timeout_ms = -1) const = 0;
 

cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/bounce/BounceConfig.h

@@ -166,7 +166,9 @@ struct BounceConfig
     int requestTimeoutMs{30000}; // request_timeout_ms; must be > 0 — the whole failure model
                                  // (abandoned-flow resolution, receiver lease, quarantine) hangs off
                                  // this timer, so applyParam rejects 0 (negatives already fail the
-                                 // strict unsigned parse) and keeps the current value.
+                                 // strict unsigned parse) and keeps the current value. It also bounds
+                                 // how long a failed request waits for the receiver's answers (after
+                                 // the failure, and again after each answer) before resolving in-doubt.
     // Receiver-side lease on granted regions — DERIVED in deriveDependentTimeouts() as
     // 2 x requestTimeoutMs, not an independent knob (tests may still set the field directly). A
     // dead sender emits neither DATA nor a cancel, which is unobservable through the protocol

cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/bounce/BounceTransferStatus.h

@@ -0,0 +1,103 @@
+/*
+ * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include "tensorrt_llm/executor/cache_transmission/nixl_utils/bounce/BounceTransport.h"
+#include "tensorrt_llm/executor/transferAgent.h"
+
+#include <chrono>
+#include <cstdint>
+#include <future>
+#include <string>
+#include <utility>
+
+namespace tensorrt_llm::executor::kv_cache::bounce
+{
+
+/// TransferStatus over the bounce transport's completion future (the value Python's wait() reads).
+/// Header-only so its contract can be unit-tested over hand-built futures.
+class BounceTransferStatus final : public TransferStatus
+{
+public:
+    explicit BounceTransferStatus(std::shared_future<BounceResult> fut)
+        : mFut(std::move(fut))
+    {
+    }
+
+    [[nodiscard]] bool isCompleted() const override
+    {
+        // Terminal AND the destination is quiesced (BounceResult::quiesced): kSUCCESS, or a kFAILURE the
+        // receiver has fully answered; an in-doubt kFAILURE stays false. Callers read the outcome from
+        // wait()/getLastStatusStr. get() cannot throw: every erase/shutdown path resolves the promise
+        // with set_value first, so broken_promise is unreachable.
+        if (!isReady())
+        {
+            return false;
+        }
+        auto const& result = mFut.get();
+        return result.state == TransferState::kSUCCESS || result.quiesced;
+    }
+
+    [[nodiscard]] TransferState wait(int64_t timeoutMs) const override
+    {
+        if (!mFut.valid())
+        {
+            return TransferState::kFAILURE;
+        }
+        if (timeoutMs < 0)
+        {
+            return mFut.get().state;
+        }
+        if (mFut.wait_for(std::chrono::milliseconds(timeoutMs)) == std::future_status::ready)
+        {
+            return mFut.get().state;
+        }
+        return TransferState::kIN_PROGRESS;
+    }
+
+    /// Failure cause recorded by the bounce transport (empty while in flight / on success). An
+    /// in-doubt failure says so, since only then may the destination still be written.
+    [[nodiscard]] std::string getLastStatusStr() const override
+    {
+        if (!isReady())
+        {
+            return {};
+        }
+        auto const& result = mFut.get();
+        if (result.reason == BounceFailReason::kNone)
+        {
+            return {};
+        }
+        std::string detail = toString(result.reason);
+        if (!result.quiesced)
+        {
+            detail += "; destination not proven quiescent";
+        }
+        return detail;
+    }
+
+private:
+    [[nodiscard]] bool isReady() const
+    {
+        return mFut.valid() && mFut.wait_for(std::chrono::seconds(0)) == std::future_status::ready;
+    }
+
+    std::shared_future<BounceResult> mFut;
+};
+
+} // namespace tensorrt_llm::executor::kv_cache::bounce