fix(storage): preserve IOPS throttle quota across re-probes

[detail-app]

Detail bug report: View on Detail

Bug

The IOPS throttle in Statistics (foyer-storage/src/io/device/statistics.rs) did not enforce configured read_iops/write_iops. Metric::throttle() stored the recomputed f64 quota as quota as isize, which in Rust truncates toward zero. With per-IO counting, a single IO produces a small fractional negative debt (-0.999…) that rounds to 0, wiping the debt on every probe.

The production read path (engine.rs:630) and the write admission path (store.rs via filter.rs) discard the returned Duration and re-poll, so the debt was cleared in O(number of probes) instead of O(time/throttle). A configured read_iops=10 admitted ~500/sec under a 1000/sec lookup rate — a ~50× violation — degrading the throttle to a coin-flip gate. Throughput (byte) throttling was largely unaffected because byte debts are large. This regression was introduced when the lock-free Metric/Statistics refactor (PR foyer-rs#1086) replaced the prior Mutex-protected, f64-quota IoThrottler.

Fix

• Store the quota losslessly as an AtomicU64 of f64::to_bits() instead of truncating to AtomicIsize.
• Update record() and throttle() with CAS loops (integer fetch_sub is meaningless on bit-encoded floats), keeping the float-quota arithmetic atomic under concurrent probes/records.
• Advance the refill timestamp with fetch_max so it never moves backward under contention.

This restores the token-bucket semantics the deleted IoThrottler had.

Testing

Committed unit tests (run by default):

• test_fractional_negative_debt_preserved_losslessly — a -0.999 quota survives a probe and stays negative (guards the root cause directly).
• test_iops_single_io_debt_survives_rapid_probes — one IO under a 1 iops/s limit stays throttled across 2000 rapid re-probes (the production poll pattern).
• test_iops_debt_releases_after_refill_window — the throttle releases after the refill window elapses (guards against over-throttle).
• test_concurrent_record_and_throttle_is_consistent — 8 threads interleaving record()/throttle() keep the quota finite and the cumulative counter exact (guards the new CAS loops).
• test_production_pattern_admission_respects_configured_limit — simulating probe→latency→record, a read_iops=1 limit admits far fewer than 1000 rapid attempts.

Committed E2E tests (#[ignore]d wall-clock rate tests against a real FsDevice + psync MonitoredIoEngine, following the repo's existing convention of ignoring throttle rate tests): test_e2e_real_read_path_low_limit_enforced and test_e2e_real_read_path_high_limit_scales. With the fix, a 10 iops/s limit caps admitted reads to ≤30 over 0.5s and a 100 iops/s limit admits ~50 — the configured rate governs. They also assert disk_read_ios() equals admitted reads, exercising the real read-completion-callback recording path.

Regression-guard check (dev-only, not versioned): I temporarily reintroduced the as isize truncation and re-ran the suite — the five unit tests and both E2E tests failed (the admission test reproduced the report's exact got 500; the E2E 10 iops/s case admitted ~7400 reads in 0.5s). Restoring the fix returned all of them to green, confirming the tests actually catch the bug.

Routine checks: unit tests, cargo check, cargo clippy, and cargo fmt --check pass for the changed file with no new warnings. The full io::* suite, the foyer-storage suite (including storage_fuzzy_test), and the umbrella foyer crate tests pass with no regressions. cargo clippy -p foyer-storage --all-targets -- -D warnings fails both before and after this change on a pre-existing, unrelated dead_code field in store.rs; I confirmed via git stash that it reproduces on the clean baseline.

---

Automatic Fixes PRs can be configured here.