feat(rest-client): resilient HTTP client with retry, circuit breaker, and correlation ID#8
Open
arjunmehta-git wants to merge 1 commit into
Open
feat(rest-client): resilient HTTP client with retry, circuit breaker, and correlation ID#8arjunmehta-git wants to merge 1 commit into
arjunmehta-git wants to merge 1 commit into
Conversation
…aker - Exponential backoff with jitter (429, 5xx, max 3 retries) - Circuit breaker via opossum (closed/open/half-open states) - Correlation ID propagation via AsyncLocalStorage + X-Correlation-ID header - GET response caching with TTL - OpenTelemetry trace header injection - Unit tests for retry and circuit breaker behavior Signed-off-by: Alex Quincy <alexquincy@users.noreply.github.com>
hortison
reviewed
May 27, 2026
hortison
left a comment
hortison
left a comment
There was a problem hiding this comment.
The circuit breaker implementation looks right. Key note on the README Mermaid diagram - the half-open state description should say 'allows ONE test request' not 'allows requests'. The whole point of half-open is a single probe to test if the downstream recovered. Otherwise teams implement it wrong.
Also: the volumeThreshold for the circuit breaker (minimum calls before the error percentage is evaluated) should be documented. Default in opossum is pretty low - you might trip the circuit on 2 failures in the first few requests at startup. Recommend setting it to at least 10.
miacycle
reviewed
May 27, 2026
miacycle
left a comment
miacycle
left a comment
There was a problem hiding this comment.
Love the correlation ID via AsyncLocalStorage pattern - this is the right way to do it. Express middleware is included which is great. One ask: add an example of how to use it with Fastify too, since we're recommending that for new services.
There was a problem hiding this comment.
Pull request overview
Adds a new rest-client/ Node.js starter that demonstrates resilient outbound HTTP usage: an Axios-based client wrapped with an opossum circuit breaker, exponential-backoff retry on 429/5xx, a TTL GET cache, AsyncLocalStorage-backed correlation ID propagation, a placeholder OTel header hook, plus a README and Jest tests. It slots into the existing starter catalog alongside kafka/ and addresses issue #5.
Changes:
- New
ResilientHttpClientwith circuit breaker, retry-with-jitter, GET TTL cache, and correlation/OTel header injection. - New
correlation-context.tsexposingAsyncLocalStorage-basedgetCorrelationIdand an ExpresscorrelationMiddleware. - README documenting state machine, retry policy, usage, and config; Jest tests covering success, cache, retry, breaker, and correlation header.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| rest-client/README.md | Starter docs: breaker state diagram, retry policy, usage, config reference. |
| rest-client/nodejs/resilient-client.ts | Core resilient HTTP client (Axios + opossum + retry + cache + OTel placeholder). |
| rest-client/nodejs/correlation-context.ts | AsyncLocalStorage correlation ID storage and Express middleware. |
| rest-client/nodejs/tests/resilient-client.test.ts | Jest tests for GET/cache/retry/breaker/correlation behaviors. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+4
| import axios, { AxiosInstance, AxiosRequestConfig, AxiosResponse } from 'axios'; | ||
| import CircuitBreaker from 'opossum'; | ||
| import { randomUUID } from 'crypto'; | ||
| import { getCorrelationId } from './correlation-context'; |
Comment on lines
+124
to
+126
| return this.withRetry(() => | ||
| this.axios.request({ method, url: path, data: body, ...requestConfig }), | ||
| ); |
Comment on lines
+42
to
+85
| private readonly cache = new Map<string, CacheEntry>(); | ||
| private readonly cacheTtl: number; | ||
|
|
||
| constructor(options: ResilientHttpClientOptions) { | ||
| this.axios = axios.create({ | ||
| baseURL: options.baseURL, | ||
| timeout: options.timeout ?? 10_000, | ||
| }); | ||
|
|
||
| this.cacheTtl = options.cache?.ttl ?? 60_000; | ||
|
|
||
| // Wrap the internal request function with a circuit breaker. | ||
| // The breaker transitions: closed (normal) -> open (failing) -> half-open (testing). | ||
| this.breaker = new CircuitBreaker(this.executeRequest.bind(this), { | ||
| errorThresholdPercentage: options.circuitBreaker?.errorThresholdPercentage ?? 50, | ||
| timeout: options.circuitBreaker?.timeout ?? 5_000, | ||
| resetTimeout: options.circuitBreaker?.resetTimeout ?? 30_000, | ||
| // Minimum calls before the error percentage is evaluated. | ||
| // Prevents tripping on 1-2 startup failures before traffic ramps up. | ||
| volumeThreshold: options.circuitBreaker?.volumeThreshold ?? 10, | ||
| }); | ||
|
|
||
| this.breaker.on('open', () => { | ||
| console.warn(`[ResilientHttpClient] Circuit breaker OPEN for ${options.baseURL}`); | ||
| }); | ||
| this.breaker.on('halfOpen', () => { | ||
| console.info(`[ResilientHttpClient] Circuit breaker HALF-OPEN - sending test request`); | ||
| }); | ||
| this.breaker.on('close', () => { | ||
| console.info(`[ResilientHttpClient] Circuit breaker CLOSED - resuming normal traffic`); | ||
| }); | ||
| } | ||
|
|
||
| async get<T>(path: string, config?: AxiosRequestConfig): Promise<T> { | ||
| const cacheKey = `GET:${path}:${JSON.stringify(config?.params ?? {})}`; | ||
| const cached = this.cache.get(cacheKey); | ||
|
|
||
| if (cached && cached.expiresAt > Date.now()) { | ||
| return cached.data as T; | ||
| } | ||
|
|
||
| const response = await this.breaker.fire('GET', path, undefined, config) as AxiosResponse<T>; | ||
|
|
||
| this.cache.set(cacheKey, { data: response.data, expiresAt: Date.now() + this.cacheTtl }); |
Comment on lines
+134
to
+160
| let lastError: Error | undefined; | ||
|
|
||
| for (let attempt = 1; attempt <= maxAttempts; attempt++) { | ||
| try { | ||
| return await fn(); | ||
| } catch (err) { | ||
| const status = (err as { response?: { status: number } }).response?.status; | ||
| const isRetryable = status === 429 || (status !== undefined && status >= 500); | ||
|
|
||
| if (!isRetryable || attempt === maxAttempts) { | ||
| throw err; | ||
| } | ||
|
|
||
| lastError = err as Error; | ||
| const baseDelay = Math.pow(2, attempt) * 100; | ||
| const jitter = Math.random() * baseDelay; | ||
| const delay = baseDelay + jitter; | ||
|
|
||
| console.warn( | ||
| `[ResilientHttpClient] Attempt ${attempt}/${maxAttempts} failed (${status}), ` + | ||
| `retrying in ${Math.round(delay)}ms`, | ||
| ); | ||
| await sleep(delay); | ||
| } | ||
| } | ||
|
|
||
| throw lastError; |
| Attempt 1: immediate | ||
| Attempt 2: ~200ms + jitter | ||
| Attempt 3: ~400ms + jitter | ||
| Attempt 4: give up, throw |
Comment on lines
+163
to
+170
| private getOtelHeaders(): Record<string, string> { | ||
| // Wire in your OpenTelemetry propagator here. | ||
| // Example with @opentelemetry/api: | ||
| // const carrier: Record<string, string> = {}; | ||
| // propagation.inject(context.active(), carrier); | ||
| // return carrier; | ||
| return {}; | ||
| } |
Comment on lines
+24
to
+32
| export function correlationMiddleware(req: Request, res: Response, next: NextFunction): void { | ||
| const correlationId = | ||
| (req.headers['x-correlation-id'] as string | undefined) ?? randomUUID(); | ||
|
|
||
| res.setHeader('x-correlation-id', correlationId); | ||
|
|
||
| correlationStorage.run({ correlationId }, () => { | ||
| next(); | ||
| }); |
Comment on lines
+1
to
+3
| # REST API Client Starter - Resilience Patterns | ||
|
|
||
| Resilient HTTP client for Node.js services calling external REST APIs. Implements retry, circuit breaker, correlation ID propagation, and response caching. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
REST API client starter demonstrating resilience patterns. Closes #5.
Patterns included
Status: Ready for review - leaving open for feedback before merge