Add cancellation and deadline support to Session.process_request

[bmehta001]

Summary

Adds cancellation and timeout support for synchronous inference without introducing a separate operation API.

• Request.Cancel() stops the active invocation and interrupts its ORT GenAI generator.
• Request timeouts start a fresh countdown for each ProcessRequest call and include chat/audio admission wait.
• Session.Cancel() permanently cancels all active and queued calls on that session.
• Distinct embeddings requests remain concurrent and can be cancelled independently.
• C#, JavaScript, and Python expose the same behavior and avoid releasing native sessions while calls are active.

Results

Cause	Result
Request cancellation	FOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED
Timeout	FOUNDRY_LOCAL_ERROR_TIMEOUT
Session cancellation	Active/queued calls are cancelled; later calls return INVALID_USAGE
Streaming callback stops consumption	Existing successful FINISH_NONE behavior

Cancelled or timed-out chat turns are not committed to history. A generator stopped through terminate_session is discarded and rebuilt from committed history on the next call. Embeddings responses remain all-or-nothing.

Assumptions

• A Request is used by one invocation at a time; sequential reuse is supported. This PR does not enforce that contract at runtime.
• Admission order is unspecified.
• No separate public operation handle is added.

Validation

• Native Debug build passes locally.
• Focused native cancellation tests pass.
• Binding lifecycle tests are included with their SDK changes.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview	Aug 15, 2026 2:28am

[Copilot]

Pull request overview

Adds cross-language cancellation and deadline support for synchronous inference, including engine-level interruption and teardown handling.

Changes:

• Adds native session cancellation, request deadlines, and timeout errors.
• Exposes cancellation/timeouts through C++, Python, C#, and JavaScript.
• Adds chat cancellation and deadline regression tests.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
sdk_v2/python/src/foundry_local_sdk/session.py	Adds timeout, cancellation, and teardown signaling.
sdk_v2/python/src/foundry_local_sdk/request.py	Adds request timeout configuration.
sdk_v2/python/src/foundry_local_sdk/_native/build_cffi.py	Extends Python C ABI definitions.
sdk_v2/js/src/session.ts	Adds AbortSignal cancellation and session cancellation.
sdk_v2/js/src/request.ts	Adds request deadlines.
sdk_v2/js/src/detail/native.ts	Extends native TypeScript interfaces.
sdk_v2/js/src/detail/errors.ts	Adds the timeout error code.
sdk_v2/js/native/src/session.h	Declares native cancellation methods.
sdk_v2/js/native/src/session.cc	Implements native session cancellation bindings.
sdk_v2/js/native/src/request.h	Declares native timeout support.
sdk_v2/js/native/src/request.cc	Implements native timeout binding.
sdk_v2/cs/src/Session.cs	Adds token/session cancellation and teardown signaling.
sdk_v2/cs/src/Request.cs	Adds TimeSpan deadlines.
sdk_v2/cs/src/Detail/NativeMethods.cs	Extends C# ABI declarations.
sdk_v2/cs/src/Detail/FoundryLocalApi.cs	Maps timeout errors and session cancellation.
sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc	Adds cancellation and deadline tests.
sdk_v2/cpp/src/inferencing/session/session.h	Defines session cancellation state and watchdog support.
sdk_v2/cpp/src/inferencing/session/session.cc	Implements cancellation, tracking, and deadlines.
sdk_v2/cpp/src/inferencing/session/session_manager.h	Documents process-wide cancellation.
sdk_v2/cpp/src/inferencing/session/session_manager.cc	Cancels sessions during shutdown.
sdk_v2/cpp/src/inferencing/session/request.h	Implements reusable request deadlines.
sdk_v2/cpp/src/inferencing/session/oga_generator_cancellable.h	Declares the OGA cancellation adapter.
sdk_v2/cpp/src/inferencing/session/oga_generator_cancellable.cc	Implements engine termination.
sdk_v2/cpp/src/inferencing/session/live_session_registry.h	Declares live-session tracking.
sdk_v2/cpp/src/inferencing/session/live_session_registry.cc	Implements the live-session registry.
sdk_v2/cpp/src/inferencing/session/cancellable.h	Defines the cancellation interface.
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.h	Adds request-aware embedding generation.
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc	Adds embedding cancellation checks.
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc	Publishes chat generators for cancellation.
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h	Makes chat generators cancellable.
sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc	Adds cancellation throughout audio generation.
sdk_v2/cpp/src/inferencing/generative/audio/audio_generator.h	Makes audio generators cancellable.
sdk_v2/cpp/src/c_api.cc	Implements new C ABI operations.
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.inline.h	Implements C++ wrapper methods.
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.h	Exposes C++ cancellation and deadlines.
sdk_v2/cpp/include/foundry_local/foundry_local_c.h	Extends the public C ABI.
sdk_v2/cpp/CMakeLists.txt	Builds the new cancellation sources.

Suppressed comments (2)

sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc:604

• This second raw OGA decode loop has the same termination-exception gap: Session::Cancel() now terminates the generator during GenerateNextToken(), but this helper does not catch the expected runtime error. A Nemotron cancellation or deadline can therefore bypass normal cancellation/timeout handling. Handle termination when original_request.ShouldStop() is true and rethrow unrelated engine failures.

  while (!generator.IsDone() && !generator.IsSessionTerminated() && !original_request.ShouldStop()) {

sdk_v2/js/src/session.ts:275

• Request_Cancel only sets the request flag; it never invokes the active generator's Cancel(). Consequently this AbortSignal still waits for an in-progress prefill/decode call and cannot interrupt a non-terminating compute as documented. Moreover, the native cancellation path returns a FINISH_NONE response rather than FOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED, so this promise resolves instead of rejecting. Associate request cancellation with its active generator and define the abort rejection before exposing this option.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 54 out of 54 changed files in this pull request and generated no new comments.

Suppressed comments (6)

sdk_v2/python/src/foundry_local_sdk/session.py:485

• _check_open() releases the lifecycle lock before this native call, so _close() can transition to closing, release the native session, and leave this thread calling Session_Cancel with None or a freed pointer. cancel() is explicitly thread-safe and can race teardown; hold an active-call lease around the native call just like process_request() does.
  sdk_v2/python/src/foundry_local_sdk/request.py:138
• A positive timeout below one millisecond is truncated to 0, which disables the deadline even though the API documents that only None and non-positive values disable it. Round positive durations up to the next millisecond so a requested deadline is never silently removed.
  sdk_v2/cs/src/Request.cs:148
• A positive TimeSpan shorter than one millisecond casts to 0, silently disabling the timeout despite the contract saying only zero or negative values disable it. Round positive durations up to one millisecond rather than truncating them.
  sdk_v2/python/src/foundry_local_sdk/session.py:480
• This documentation omits the method's terminal behavior: native Session_Cancel permanently marks the session cancelled, so even an idle call causes every later processing call to fail with invalid usage. State that explicitly; describing only interruption and saying it is safe while idle can lead callers to expect the session remains reusable.
  sdk_v2/cs/src/Session.cs:226
• This public contract does not mention that Cancel() is terminal. The native implementation permanently cancels the session, and subsequent processing calls fail with invalid usage even when Cancel() was invoked while idle. Document that behavior so callers do not expect this to be a reusable, current-operation-only cancellation.
  sdk_v2/js/src/session.ts:308
• The documentation presents this as interruption of current work, but native Session::Cancel() is terminal: calling it while idle still makes all later processing calls fail with FlErrorCode.InvalidUsage. Document the permanent state transition so consumers do not attempt to reuse the session.

[Copilot]

Pull request overview

Copilot reviewed 54 out of 54 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

sdk_v2/js/src/session.ts:285

• This listener has the same queueing race as streaming: an abort can occur before the async worker attaches the request's invocation state, and native Request.Cancel() then intentionally does nothing. Since the listener runs only once, the aborted operation can consume the entire generation before line 289 merely converts its successful result to AbortError. Retry until settlement or have the addon latch cancellation for the queued worker.
  sdk_v2/js/src/session.ts:166
• Request.Cancel() is now an idle no-op, so this one-shot abort can race the queued native worker: if the signal fires after processStreamingRequest returns but before BeginInvocation attaches cancellation state, cancellation is lost and the full inference still runs before JavaScript rejects it. Keep retrying cancellation until the native promise settles, or add a pending-cancellation handshake in the addon.

This issue also appears on line 285 of the same file.
sdk_v2/cs/src/Session.cs:356

• This catch wraps native TimeoutException and direct Request.Cancel()'s OperationCanceledException in FoundryLocalException whenever the linked token itself was not canceled. That makes streaming expose different error types from non-streaming and hides the newly added timeout result. Preserve these two mapped exception types before applying the generic streaming wrapper.

[Copilot]

Pull request overview

Copilot reviewed 54 out of 54 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk_v2/cpp/src/inferencing/session/live_session_registry.cc:18

• There is still a shutdown admission race here: a Session::Create that passes IsShutdownRequested() immediately before shutdown can call Add after CancelAll() has already snapshotted the registry, so its control is never terminated and the session can keep running after shutdown. Make registry shutdown a latched operation synchronized with Add (for example, a TerminateAll that marks the registry terminal under this mutex, with later additions rejected or immediately terminated).

void LiveSessionRegistry::Add(const std::shared_ptr<SessionControl>& control) {
  std::lock_guard<std::mutex> lock(mutex_);
  controls_.emplace_back(control);

sdk_v2/js/native/src/session.cc:394

• CancelOn catches foundry_local::Error as a generic std::exception, so failures lose the stable FoundryLocalError name and code that every other addon entry exposes. Route this call through CallChecked so callers can reliably distinguish native error codes.