feat: ETCD high availability client failover - lease keep-alive resilience

[kthui]

Overview:

When a replica of ETCD servers are available, the client should automatically failover to the other ETCD server when the connection to the current server is lost, providing resilience to any single ETCD server failure.

Details:

• Lease keep-alive failover ✅
• Lease watcher failover 🚧 - will be included in the next PR

Tests:

• Basic aggregated serving ETCD failover ✅ - need to remove allowed to fail on next PR
• Graceful shutdown when ETCD failover failed ✅

Where should the reviewer start?

Start with the test cases, and then move on to the new connector.rs, which substitutes the etcd_client::Client in etcd.rs, and then the other Rust pieces.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to [BUG]: Add fault tolerance to our etcd WatchClient #1592
• Relates to [BUG]: ETCD lease lost, leading to series of events that does not auto-recover. #3493

[rmccorm4]

@coderabbitai review

[kthui]

@coderabbitai review

it disappeared from the checks

[Copilot]

Pull Request Overview

This PR implements ETCD high availability support for lease keep-alive operations, enabling automatic client failover when an ETCD server becomes unavailable. The implementation introduces a Connector abstraction that manages ETCD client connections and handles reconnection logic with exponential backoff.

Key changes:

• Implemented Connector for managing ETCD client connections with automatic reconnection
• Enhanced keep_alive function to detect stream failures and trigger reconnection
• Updated all ETCD client access points to use the new async-aware connection management

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/fault_tolerance/etcd_ha/utils.py	Test utilities for ETCD HA scenarios including cluster management and process monitoring
tests/fault_tolerance/etcd_ha/test_vllm.py	Integration tests verifying ETCD failover behavior and graceful shutdown
lib/runtime/src/transports/etcd/connector.rs	New connector module managing ETCD connections with reconnection logic
lib/runtime/src/transports/etcd/lease.rs	Updated lease keep-alive to use connector and handle stream reconnection
lib/runtime/src/transports/etcd.rs	Refactored client to use connector and provide async client access
lib/runtime/src/transports/etcd/lock.rs	Updated to use async client accessor
lib/runtime/src/storage/key_value_store/etcd.rs	Updated to use async client accessor

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.