Repository navigation
fix(cache): GetStatusKeys returns cache keys, not raw node IDs - #19
Conversation
The status and snapshot maps are keyed by hash.ID(node): CreateWatch and CreateDeltaWatch compute nodeID := cache.hash.ID(request.GetNode()) and getOrCreateStatus stores shard.status[nodeID]. GetStatusKeys threw those map keys away - allStatus() returns only the values - and reported statusInfo.GetNode().GetId(), the raw node ID off the stored proto. With IDHash the two are the same string, so nothing looked wrong. For any NodeHash that derives a key from more than node.Id the list is unusable: GetSnapshot misses, SetSnapshot/UpsertResources write to a key no watch is registered against, and both fail silently because a missing snapshot is not an error condition on those paths. Callers cannot work around it, since the correct key exists only inside the cache. Found in ShareChat/nexus#920: xlr8's optional per-pod cache key appends a pod suffix in IDHash.ID, and every xlr8 path that feeds GetStatusKeys() into a cache lookup - snapshot fan-out, the xDS drift reconciler, /config_lag - silently addressed the wrong slot the moment that flag was enabled. The reconciler in particular would have reported zero drift forever while looking perfectly healthy. GetStatusKeys now walks the shards and collects the map keys, which is what its doc comment ("all node IDs in the status map") already described. It also drops the intermediate allStatus() slice. Behaviour is unchanged for IDHash, so this is a no-op for existing callers. TestGetStatusKeysAreUsableAsCacheKeys uses a hash with a suffix and fails without the fix with: GetStatusKeys returned "node-a", want the cache key "node-a~pod-1" Pre-existing failures in this package (TestSnapshotCacheWatch, TestSnapshotCacheDeltaWatch, TestSnapshotDeltaCacheWatchTimeout, TestSnapshotCacheWithTTL) are identical before and after this change.
|
Understand this PR’s impact Explore downstream dependencies and potential security impact with Blast Radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: ShareChat/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (2)
Included review availability: 4 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour. 📝 WalkthroughWalkthrough
ChangesStatus cache key correction
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
The bug
The status and snapshot maps are keyed by
hash.ID(node):GetStatusKeysthrew those map keys away —allStatus()returns only the values — and reportedstatusInfo.GetNode().GetId(), the raw node ID off the stored proto.With
IDHashthe two are the same string, so nothing ever looked wrong. For anyNodeHashthat derives a key from more thannode.Id, the returned list is unusable:GetSnapshot(key)misses — it indexesshard.snapshots[node]with no re-hashSetSnapshot/UpsertResourceswrite to a key no watch is registered againstCallers cannot work around it either, since the correct key exists only inside the cache.
How it was found
ShareChat/nexus#920. xlr8 has an optional per-pod cache key that appends a pod suffix in
IDHash.ID. Every xlr8 path that feedsGetStatusKeys()into a cache lookup — snapshot fan-out, the xDS drift reconciler, the/config_lagendpoint — silently addressed the wrong slot the moment that flag was enabled. The reconciler would have reported zero drift forever while looking perfectly healthy, and the preprod test plan that was meant to validate the flag gated on generation cost staying flat, which it would have, precisely because nothing was reaching any proxy.Credit to @jensoncs for tracing it to this function.
The fix
Walk the shards and collect the map keys — which is what the doc comment ("all node IDs in the status map") already described. Also drops the intermediate
allStatus()slice.Behaviour is unchanged for
IDHash, so this is a no-op for existing callers.Testing
TestGetStatusKeysAreUsableAsCacheKeysuses a hash with a suffix. Without the fix:Existing tests assert only
len(keys), so none depended on the old values.Pre-existing failures in this package —
TestSnapshotCacheWatch,TestSnapshotCacheDeltaWatch,TestSnapshotDeltaCacheWatchTimeout,TestSnapshotCacheWithTTL— are identical before and after this change, verified by stashing.pkg/test/mainalso fails to build on cleannexusfor an unrelated reason (VTMarshaledResourcedoes not implementtypes.Resource).Summary by CodeRabbit
Bug Fixes
Tests