Skip to content

fix: prevent race condition causing stale middleware chain during concurrent provider updates - #21

Open
sonjaq wants to merge 1 commit into
jahmeergnlt:mainfrom
sonjaq:fix/race-condition-stale-middleware-chain
Open

fix: prevent race condition causing stale middleware chain during concurrent provider updates#21
sonjaq wants to merge 1 commit into
jahmeergnlt:mainfrom
sonjaq:fix/race-condition-stale-middleware-chain

Conversation

@sonjaq

@sonjaq sonjaq commented Aug 7, 2026

Copy link
Copy Markdown

Summary

Fixes the race condition described in #1 where concurrent configuration updates from multiple providers (File provider, Kubernetes CRD provider) can cause the HTTP/TCP handler chain to execute using a stale middleware chain from a previous configuration.

Root Cause

Two bugs were identified:

1. atomic.Value type panic

The initial handler in NewServer() was stored as http.HandlerFunc, but switchConfigs() stored *http.ServeMux — these are different concrete types. Go's sync/atomic.Value panics when Store is called with a different type than the first store. Under concurrent provider loads, this panic would crash the server.

Fix: Store *http.Handler (pointer to interface) so every Store call uses the same concrete type regardless of the underlying handler implementation.

2. Non-atomic handler chain swap

While the sync.RWMutex in switchConfigs serialized config updates, the handler chain was being constructed and swapped with a type mismatch that could panic under load. The fix ensures the entire handler chain (router + middleware) is built from a single immutable configuration snapshot within the lock, then atomically swapped as one unit.

Approach

  • EntryPoint.handler now stores *http.Handler (pointer to interface) instead of a raw interface, ensuring type consistency across all atomic.Value.Store calls
  • The switchConfigs method builds the complete mux with all middleware within the existing s.mu.Lock() scope, then swaps atomically — no window for partial state
  • The initial handler in NewServer() is stored with the same *http.Handler wrapper

Testing

  • Enhanced TestConcurrentConfigurationUpdates with 4 concurrent request workers (2000 total requests) and an atomic violation counter to detect any middleware/response mismatches
  • Added TestConcurrentConfigurationUpdatesWithRaceDetector — 500 rapid config toggles under concurrent reads to stress the atomic swap
  • All tests pass with go test -race -count=5, zero flakiness, zero race detector warnings
go test -race -count=5 -v ./...

Team

This PR was collaboratively developed by Sonja Leaf (@sonjaq) and Cliff (AI agent, Hermes by Nous Research).

Cliff handled code analysis, implementation, and testing under Sonja's direction and review.

/claim #1

…current provider updates

Two bugs fixed:

1. atomic.Value type panic: The initial handler was stored as http.HandlerFunc
   but switchConfigs stored *http.ServeMux — different concrete types cause
   sync/atomic to panic. Fixed by storing *http.Handler (pointer to interface)
   so every Store call uses the same concrete type.

2. Race condition on concurrent config updates: While the mu lock serializes
   switchConfigs calls, the atomic.Value Store was using inconsistent types
   which could panic under concurrent provider loads. The handler chain
   (router + middleware) is now built entirely within the lock from a single
   immutable configuration snapshot, then atomically swapped as one unit.

Tests:
- Enhanced TestConcurrentConfigurationUpdates with 4 concurrent request workers
  and violation counter to detect middleware/response mismatches
- Added TestConcurrentConfigurationUpdatesWithRaceDetector for high-frequency
  config toggling under concurrent reads
- All tests pass with go test -race -count=5, zero flakiness

Co-authored-by: Sonja Leaf <sonjaq@users.noreply.github.com>
Co-authored-by: Cliff <cliff@hermes.ai>
@opirebot

opirebot Bot commented Aug 7, 2026

Copy link
Copy Markdown

👀 We've notified the reward creators here.
Make sure your payment account is ready to receive the payment for your hard work 💪

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant