monad-wireauth: split handshake rate limits by cookie validity

[dshulyak]

right now the "cookie/ip validation" path only kicks in once
total_sessions >= low_watermark_sessions. an attacker can try to keep
total_sessions just below that threshold while still generating enough
handshake traffic to consume the global handshake budget, causing honest
peers new connections to get delayed or dropped without ever triggering
cookie-gated, per-ip controls.

split the handshake budget into two independent rate limits:

• handshake_cookie_unverified_rate_limit for cookie-invalid initiations
• handshake_cookie_verified_rate_limit for cookie-valid initiations

once the cookie-unverified budget is exhausted we send cookie replies
instead of dropping, so honest peers can obtain a cookie and retry into
the protected verified lane. cookie replies are smaller than initiation/response,
so even if we reply under attack we are not amplifying traffic

closes: #2777

[Copilot]

Pull request overview

This PR addresses a security vulnerability in the handshake rate limiting mechanism by splitting the global handshake budget into two independent rate limits based on cookie validity. Previously, an attacker could exhaust the single global handshake budget while keeping total_sessions below the low watermark, effectively preventing legitimate new connections without triggering cookie-based protections.

Changes:

• Split handshake rate limit into separate budgets for cookie-verified and cookie-unverified handshake initiations
• When unverified budget is exhausted, send cookie replies (if verified budget permits) instead of silently dropping, allowing honest peers to obtain cookies
• When both budgets are exhausted, drop requests to prevent amplification attacks

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
monad-wireauth/src/config/mod.rs	Replaced single handshake_rate_limit field with handshake_cookie_unverified_rate_limit and handshake_cookie_verified_rate_limit; removed unused max_requests_per_ip field
monad-wireauth/src/filter.rs	Implemented dual counter system with separate logic for verified/unverified budgets; counters are checked before watermark checks to prevent the attack
monad-wireauth/src/api.rs	Updated Filter::new() call to pass both rate limit parameters
monad-wireauth/tests/tests.rs	Updated tests to use new config fields and verify new behavior (4 packets including cookie reply instead of 2 drops)
monad-wireauth/README.md	Updated documentation to reflect new rate limits and removed deprecated max_requests_per_ip parameter

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[robertkiel]

Doesn't this reveal some of the internal state?

There are three observable outcomes, namely

• a handshake reply
• a cookie
• nothing, packet gets dropped

As an adversary, we can

• sample a random cookie
• use a previously retrieved cookie

To see if the node is busy, we send a random value. If the node replies with a cookie, we know that usage is very low and take the cookie. Otherwise, we use a correct cookie. If the packet gets dropped and we don't receive a reply, then usage has execeeded self.handshake_cookie_verified_rate_limit. If we receive a reply with the handshake reply, we know that usage is below self.handshake_cookie_verified_rate_limit

[dshulyak]

do you see a specific problem? this should not be an important factor in preventing exploit that i am trying to mitigate

cookie can be used to rate limit by ip, so to consume whole verified budget someone will need to get ips to bypass that rate limit. with default params that would be 10_000 ips

[robertkiel]

True, it does not affect the mitigation of the vulnerability but it might create a new side channel that leaks useful data for DoS attacks.

[dshulyak]

Run actions/checkout@v4
/usr/bin/docker exec  bc8335f374ace18483ba311602f79149a00e6dc944952c39534df3d1e071eeb9 sh -c "cat /etc/*release | grep ^ID"
Syncing repository: category-labs/monad-bft
Getting Git version info
Temporarily overriding HOME='/__w/_temp/e4b93360-bf33-432a-a698-ace48bb2f818' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /__w/monad-bft/monad-bft
Deleting the contents of '/__w/monad-bft/monad-bft'
Error: File was unable to be removed Error: EACCES: permission denied, unlink '/__w/monad-bft/monad-bft/target/.rustc_info.json'