Implement functional cancellation for ONNX backend inference operations

[sanchitmonga22]

Overview

Attribute	Value
Impact	High
Priority	P1
Effort	Medium
Affected Components	ONNX Backend (STT, TTS, Text Generation, Diarization)

Problem Statement

The cancel() methods in all ONNX capability classes correctly set the cancel_requested_ atomic flag, but this flag is never checked during actual inference operations. This makes cancellation non-functional - users can call cancel() but long-running operations will not be interrupted.

Why It Matters

• Long-running STT transcriptions cannot be cancelled
• TTS synthesis cannot be interrupted
• Poor user experience when operations are stuck
• Resource waste when user wants to abort

Current State

Files Affected

• src/backends/onnx/onnx_backend.cpp
• src/backends/onnx/onnx_backend.h

Cancel Methods (Correctly Implemented)

All four capability classes have cancel() methods that set the flag:

// ONNXTextGeneration::cancel() (line 198-200)
void ONNXTextGeneration::cancel() {
    cancel_requested_ = true;
}

// ONNXSTT::cancel() (line 719-721)
void ONNXSTT::cancel() {
    cancel_requested_ = true;
}

// ONNXTTS::cancel() (line 1001-1003)
void ONNXTTS::cancel() {
    cancel_requested_ = true;
}

// ONNXDiarization::cancel() (line 1125-1127)
void ONNXDiarization::cancel() {
    cancel_requested_ = true;
}

Flag Declaration (Correctly Atomic)

std::atomic<bool> cancel_requested_{false};

Problem: No Checks During Inference

The following methods perform long-running operations but never check cancel_requested_:

Method	Lines	Issue
ONNXSTT::transcribe()	480-560	No cancellation check during audio processing
ONNXSTT::decode()	635-672	No cancellation check before/after inference
ONNXTTS::synthesize()	926-982	No cancellation check before TTS generation
ONNXTextGeneration::generate()	182-191	TODO placeholder (not implemented yet)
ONNXDiarization::diarize()	1109-1113	TODO placeholder (not implemented yet)

Proposed Solution

Add cancellation checks at strategic points in inference methods:

1. ONNXSTT::transcribe() - Add check after decoding

// After line 512: SherpaOnnxDecodeOfflineStream()
if (cancel_requested_) {
    cancel_requested_ = false;  // Reset for next use
    result.error = "Operation cancelled";
    return result;
}

2. ONNXSTT::decode() - Add check before inference

// Before line 653: SherpaOnnxDecodeOfflineStream()
if (cancel_requested_) {
    cancel_requested_ = false;
    return "";
}

3. ONNXTTS::synthesize() - Add check before TTS generation

// Before line 953: SherpaOnnxOfflineTtsGenerate()
if (cancel_requested_) {
    cancel_requested_ = false;
    result.error = "Operation cancelled";
    return result;
}

4. Streaming methods - Check in callbacks

For streaming operations, the cancellation check should happen in the streaming callback to allow mid-stream cancellation.

Implementation Plan

• Add cancellation check in ONNXSTT::transcribe() before decoding
• Add cancellation check in ONNXSTT::decode() before inference call
• Add cancellation check in ONNXTTS::synthesize() before TTS generation
• Reset cancel_requested_ to false after handling cancellation
• Add unit tests for cancellation behavior
• Document cancellation behavior in API docs

Success Criteria

• Calling cancel() during transcribe() returns early with appropriate error
• Calling cancel() during synthesize() returns early with appropriate error
• Cancellation does not leave backend in inconsistent state
• cancel_requested_ is properly reset after cancellation
• No memory leaks when operations are cancelled

Related

• PR ONNX Backend - iOS - STT + TTS #1 Review Comment: "cancel_requested_ flag is set but never checked during inference, making cancellation non-functional"

---

🤖 Generated with Claude Code