[Feature]: Implement throttling resilience for large-tenant deployments (thousands of subscriptions)

[MatthewGrimshaw]

What problem are you trying to solve?

When running the Azure FinOps agent against a production tenant with thousands of subscriptions, users experience widespread HTTP 429 (Too Many Requests) errors from Azure Resource Manager (ARM) and Azure Resource Graph (ARG). This makes the agent unreliable at scale.

Root cause analysis based on the current codebase:

BulkAzureRequest in AzureQueryTools.cs uses client-side parallelism — it fires up to 50 concurrent HTTP requests from the application via Parallel.ForEachAsync, each hitting https://management.azure.com individually. This rapidly exhausts the ARM quota of 12,000 reads/hour per subscription per security principal and triggers 429 responses.

SendWithRetryAsync in HttpHelper.cs has minimal retry logic — only 3 attempts on 429, with a simplistic backoff of (attempt+1) * 5s capped at 30s. It reads Retry-After but does not read the ARM rate-limit headers (x-ms-ratelimit-remaining-subscription-reads, x-ms-ratelimit-remaining-tenant-reads) or the ARG quota headers (x-ms-user-quota-remaining, x-ms-user-quota-resets-after). This means the code cannot proactively slow down before hitting the limit.

No use of the ARG GET/LIST API — all resource queries go through the standard ARM endpoint. ARM has a hard limit of 12,000 reads/hour/subscription. The ARG GET/LIST API provides a separate quota of 4,000 requests/minute per user per subscription by routing through Azure Resource Graph, which is purpose-built for high-volume read scenarios.

No use of the Azure Batch API — instead of the server-side batch endpoint (POST https://management.azure.com/batch?api-version=2020-06-01) which processes requests in parallel on Azure's infrastructure, the code manages parallelism locally with all the associated problems: connection pooling overhead, rate-limit coordination across concurrent threads, and thundering-herd effects when quota resets.

No query grouping — when tools like AnomalyTools.cs or IdleResourceTools.cs need data across many subscriptions, they issue individual requests per subscription rather than grouping subscriptions in batches of 100–300 as recommended by Microsoft's throttling guidance.

No query staggering — all requests fire as fast as possible, causing bursty traffic patterns that are more likely to trigger throttling than evenly distributed requests.

Proposed solution

A multi-layered approach targeting HttpHelper.cs, AzureQueryTools.cs, and all tool classes that make ARM calls.

Layer 1: Enhance SendWithRetryAsync with full throttle-awareness

File: src/Dashboard/Infrastructure/HttpHelper.cs

• After every response, read and track these headers:
  • Retry-After (already partially done — keep)
  • x-ms-ratelimit-remaining-subscription-reads (ARM)
  • x-ms-ratelimit-remaining-tenant-reads (ARM)
  • x-ms-user-quota-remaining (ARG)
  • x-ms-user-quota-resets-after (ARG)
• When x-ms-ratelimit-remaining-subscription-reads drops below a configurable threshold (e.g. 100), proactively delay subsequent requests by the x-ms-user-quota-resets-after duration before continuing.
• Increase retry attempts from 3 to 5 for 429 responses.
• Add jitter to backoff delays to prevent thundering herd: delay = retryAfter * (1 + random(0, 0.5)).
• Log throttling events with telemetry tags: remaining quota, reset time, attempt number.

Reference: ARM request limits and throttling

Layer 2: Replace client-side parallel with Azure Batch API

File: src/Dashboard/AI/Tools/AzureQueryTools.cs — BulkAzureRequest method

Replace the current Parallel.ForEachAsync implementation with the Azure Batch API:

POST https://management.azure.com/batch?api-version=2020-06-01

Request structure:

{
  "requests": [
    { "httpMethod": "GET", "name": "req-0", "url": "/subscriptions/{id}/providers/..." },
    { "httpMethod": "GET", "name": "req-1", "url": "/subscriptions/{id}/providers/..." }
  ]
}

Implementation details:

• Use a batch size of 15 requests per batch call to avoid the 201 Accepted async pattern (which triggers when payloads are too large).
• Process batch chunks sequentially with throttle-aware delays between chunks.
• Check individual httpStatusCode in each response item (the outer batch call returns 200 even if individual requests fail).
• Retry individual failed requests (429s) from the batch response in a subsequent batch.
• This moves parallelism to Azure's infrastructure, eliminating local connection pooling and rate-limit coordination overhead.

Reference: Azure Batch API blog post

Layer 3: Offload GET/LIST to Azure Resource Graph

File: src/Dashboard/Infrastructure/HttpHelper.cs or src/Dashboard/AI/Tools/AzureQueryTools.cs

For eligible GET and LIST API calls against management.azure.com, append useResourceGraph=true as a query parameter to route the request through the ARG backend:

GET https://management.azure.com/subscriptions/{id}/providers/{ns}/{type}?api-version={v}&useResourceGraph=true

Implementation details:

• Create an ArmClientOptions with a custom HttpPipelinePolicy that adds useResourceGraph=true to GET and LIST calls, following the SDK sample code from Microsoft.
• Alternatively, add the query parameter in SendWithRetryAsync when the method is GET and the URL targets management.azure.com with a path matching resource listing patterns.
• Implement hybrid fallback: if ARG returns HTTP 422 (unprocessable) or 404 (not yet indexed), retry without the useResourceGraph=true flag to fall back to the Resource Provider.
• ARG GET/LIST provides 4,000 requests/minute per user per subscription — a separate, much higher quota than ARM's 12,000/hour.

Reference: ARG GET/LIST API

Layer 4: Subscription grouping and query staggering

Files: All tool files that iterate over subscriptions — AnomalyTools.cs, IdleResourceTools.cs, and any tool that queries across subscriptions.

• When querying Resource Graph across multiple subscriptions, group them into batches of 100–300 subscriptions per query rather than one query per subscription.
• Stagger batch execution with configurable delays between batches (e.g. distribute across 5-second windows) instead of firing all at once.
• For paginated results, handle $skipToken continuation and respect throttle headers between pages.

Reference: Guidance for throttled requests — Grouping queries

Files to Modify

File	Change
src/Dashboard/Infrastructure/HttpHelper.cs	Add rate-limit header tracking, proactive backoff, jitter, increased retries, optional useResourceGraph=true injection
src/Dashboard/AI/Tools/AzureQueryTools.cs	Replace BulkAzureRequest client-side parallelism with Azure Batch API; add subscription grouping
src/Dashboard/AI/Tools/AnomalyTools.cs	Use grouped/batched queries for multi-subscription anomaly detection
src/Dashboard/AI/Tools/IdleResourceTools.cs	Use grouped/batched queries for idle resource scanning
src/Dashboard/AI/Tools/HealthTools.cs	Apply throttle-aware patterns if multi-subscription
src/Dashboard/AI/Tools/ScoreTools.cs	Apply throttle-aware patterns if multi-subscription

Acceptance Criteria

• SendWithRetryAsync reads x-ms-ratelimit-remaining-subscription-reads and x-ms-user-quota-remaining from all ARM/ARG responses and proactively delays when remaining quota is low
• SendWithRetryAsync retries up to 5 times on 429 with jittered exponential backoff
• BulkAzureRequest uses POST https://management.azure.com/batch?api-version=2020-06-01 with batch size ≤15 instead of Parallel.ForEachAsync
• GET/LIST calls to ARM include useResourceGraph=true with hybrid fallback on 422/404
• Multi-subscription queries group subscriptions in batches of 100–300
• Throttling events are logged via OpenTelemetry with remaining quota, reset time, and attempt metadata
• Agent operates reliably against a tenant with 1000+ subscriptions without sustained 429 errors

References

• ARG GET/LIST API — offload reads to Resource Graph
• ARM request limits and throttling
• Guidance for throttled requests in Azure Resource Graph
• Azure Batch API for parallel management operations

Area

New AI tool (Azure / Graph / Log Analytics)

Alternatives considered

1. Increase ARM quota via support request

ARM's 12,000 reads/hour limit can be increased by contacting Microsoft support. Rejected because this is a per-tenant administrative action that cannot be enforced for all users of the agent, and it only delays the problem rather than solving the architectural root cause.

2. Client-side parallelism with smarter rate limiting (current approach, enhanced)

Keep Parallel.ForEachAsync but add a SemaphoreSlim-based rate limiter that tracks remaining quota from response headers and gates new requests. Rejected because it still sends individual HTTP requests from the client, incurring network round-trip overhead per request, and doesn't benefit from Azure's server-side routing optimizations. The Azure Batch API achieves the same parallelism with a single HTTP round-trip per batch.

3. Azure SDK ArmClient with built-in retry policies

Replace raw HttpClient calls with the Azure SDK's ArmClient, which has built-in retry policies and respects Retry-After. Partially adopted — the ARG GET/LIST integration uses the ArmClientOptions + HttpPipelinePolicy pattern from the SDK documentation. However, fully migrating all API calls to ArmClient is a larger refactor than necessary to solve the throttling issue and would change the tool architecture significantly. The existing HttpHelper.SendWithRetryAsync pattern is a reasonable abstraction that can be enhanced in-place.

4. Caching responses locally

Cache ARM responses in-memory or in Redis to avoid repeated calls for the same resource data within a time window. Deferred — this is a valid optimization but orthogonal to the throttling fix. It should be considered as a follow-up enhancement. Resource data freshness requirements vary by tool (cost data can be cached for hours, resource health data should be near-real-time), so a generic caching layer needs careful design.

5. Using Azure Resource Graph queries exclusively (KQL)

Replace all ARM GET/LIST calls with ARG KQL queries (Resources | where type == '...'). Partially adopted — the agent already has Resource Graph query capabilities. However, ARG KQL has its own throttling limits (15 queries per 5-second window), doesn't cover all resource properties (e.g. runtime status, cost data), and has eventual consistency (seconds of delay after mutations). The useResourceGraph=true flag on GET/LIST calls is a better fit because it preserves the existing API contract while gaining ARG's higher quota.

6. Queue-based async processing

Offload all Azure queries to a background queue (Azure Queue Storage or Service Bus) with a worker that respects rate limits. Rejected because it fundamentally changes the user experience from synchronous chat responses to async notifications, adds infrastructure complexity (queue + worker), and the latency increase would make the conversational AI experience unusable.