Skip to content

fix(recovery): recover from poisoned mutex in CircuitBreaker#1338

Open
ashnaaseth2325-oss wants to merge 1 commit intomofa-org:mainfrom
ashnaaseth2325-oss:fix/circuit-breaker-mutex-poison-v2
Open

fix(recovery): recover from poisoned mutex in CircuitBreaker#1338
ashnaaseth2325-oss wants to merge 1 commit intomofa-org:mainfrom
ashnaaseth2325-oss:fix/circuit-breaker-mutex-poison-v2

Conversation

@ashnaaseth2325-oss
Copy link
Contributor

SUMMARY

This PR prevents runtime panics in the CircuitBreaker by safely handling poisoned mutex locks instead of unwrapping them. It updates lock acquisition logic across crates/mofa-foundation/src/recovery.rs, ensuring the circuit breaker continues functioning even after a thread panic.


FIX

  • Root cause:
    std::sync::Mutex::lock().unwrap() panics when the mutex is poisoned, causing cascading failures in all subsequent calls.

  • Technical approach:
    Replace .unwrap() with .unwrap_or_else(|e| e.into_inner()) to recover the guard from a poisoned mutex and continue execution safely.

Before (problematic pattern):

let mut guard = self.state.lock().unwrap();

After (safe handling):

let mut guard = self.state.lock().unwrap_or_else(|e| e.into_inner());
  • Applied consistently across:

    • state()
    • call()
    • record_success()
    • record_failure()
    • reset()

VERIFICATION

  1. Create a CircuitBreaker instance.
  2. Simulate a panic while holding the mutex (e.g., inject a panic inside a guarded section like record_failure()).
  3. After the panic, invoke methods like state() or call().

Replace .lock().unwrap() with .lock().unwrap_or_else(|e| e.into_inner())
in all five CircuitBreaker methods so a thread panic while holding the
lock does not cascade into panics on every subsequent caller.

Signed-off-by: ashnaaseth2325-oss <ashnaaseth2325@gmail.com>
@ashnaaseth2325-oss
Copy link
Contributor Author

Hello @LuigiGonnella @mugiwaraluffy56 ,
This PR fixes panic on poisoned mutex in CircuitBreaker by safely recovering the lock. It prevents cascading failures and keeps the circuit breaker working reliably.
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant