Skip to content

Commit a96a6c6

Browse files
committed
local integration tests: cover callback handlers, fetch, metrics-only, payload capture
Four usage modes customers run today had no end-to-end coverage on any surface (L2 local RIE, or the old AWS suite); PR 2a reached parity with the old suite, and these are the first items beyond it (analysis: nodejs/pr2b-coverage-analysis.md, roadmap: PR 2b). All goldens are captured from the current pre-migration implementation. - manual-callback: callback-style (event, context, callback) handler under manual datadog() wrap. The migration spike broke exactly this seam (the tracePromise wrapper replaced promisifiedHandler's call site, so callback handlers returned null to API Gateway); handler.spec.ts pins the units, this case pins the path end to end through the RIE invoke. - cjs-fetch-requests: global fetch variant of cjs-http-requests. On Node 18+ fetch is undici, instrumented by dd-trace's undici plugin — a different injection path than the http plugin that patches axios. The golden pins the injected x-datadog-* headers via the mock echo (18/18 requests carry all four headers). - manual-metrics-only: DD_TRACE_ENABLED=false — the metrics-only customer configuration. The golden pins that enhanced + custom metrics still flush while no aws.lambda span, no trace JSON, and no dd.trace_id log correlation appear. Return-value goldens are byte-identical to manual-send-metrics (verified), confirming the toggle changes only the tracing surface. - cjs-capture-payload: DD_CAPTURE_LAMBDA_PAYLOAD=true in redirect mode; the golden pins function.request / function.response span tags with the captured payloads. Also: - README: case table entries for the four cases, and a "deliberately not covered locally" list in the emulation-gaps section (streaming / time_to_first_byte, direct-API and KMS/Secrets metric paths, aws-sdk client spans, durable execution) with each item's owning suite, so the scoping discussion does not recur. - migration_parity.md: point the callback and payload-capture rows at the new goldens; add dd-core ownership rows for fetch/undici injection and Lambda profiling (DD_PROFILING_ENABLED was a documented feature with no ledger row — capability lives in dd-trace's profiler, covered by its own suites plus the L3 Profiling row). Verified: all four cases green in verify mode on node22, node18 and (manual-callback) node26 legs against shared goldens — no per-runtime overrides needed.
1 parent 2eec92d commit a96a6c6

18 files changed

Lines changed: 3485 additions & 2 deletions
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
// Callback-style handler — (event, context, callback) instead of async —
2+
// manually wrapped with datadog(). The migration spike broke exactly this
3+
// seam: the new host's tracePromise wrapper replaced promisifiedHandler's
4+
// call site, so callback handlers returned null to API Gateway. handler.spec
5+
// pins the unit behavior; this case pins the same path end to end through
6+
// the RIE invoke. The setTimeout makes the completion genuinely asynchronous
7+
// so the wrapper cannot mistake it for a sync return.
8+
const { datadog } = require("datadog-lambda-js");
9+
10+
function handle(event, context, callback) {
11+
setTimeout(() => {
12+
callback(null, { message: "hello, dog!" });
13+
}, 10);
14+
}
15+
16+
module.exports.handle = datadog(handle);
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
// fetch variant of http-requests.js. On Node 18+ the global fetch is
2+
// undici, which dd-trace instruments through its undici plugin — a different
3+
// injection path than the http/https plugin that patches axios in
4+
// http-requests.js. Both paths must keep injecting the x-datadog-* trace
5+
// context; this case pins the fetch one.
6+
//
7+
// Like http-requests.js this entry point is UNWRAPPED and runs through the
8+
// npm redirect entry (dist/handler.handler) in the cjs-fetch-requests case,
9+
// so dd-trace initializes before the user handler loads and the tracer's
10+
// undici plugin does the injection. The mock echoes the request headers and
11+
// this handler logs them, so the golden shows the injected downstream trace
12+
// context (x-datadog-trace-id / x-datadog-parent-id, normalized to XXXX).
13+
const { sendDistributionMetric } = require("datadog-lambda-js");
14+
15+
const urls = (process.env.MOCK_HTTP_URLS ||
16+
"https://ip-ranges.datadoghq.com,https://ip-ranges.datadoghq.eu"
17+
).split(",");
18+
19+
async function handle(event, context) {
20+
const responsePayload = { message: "hello, dog!" };
21+
22+
sendDistributionMetric("serverless.integration_test.execution", 1, "function:fetch-request");
23+
24+
for (let index = 0; index < urls.length; index++) {
25+
const response = await fetch(urls[index]);
26+
const body = await response.json();
27+
console.log(`mock-http saw headers for ${urls[index]}: ${JSON.stringify(body.headers)}`);
28+
}
29+
30+
console.log(`Snapshot test fetch requests successfully made to URLs: ${urls}`);
31+
32+
return responsePayload;
33+
}
34+
35+
module.exports.handle = handle;

‎integration_tests_local/README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,12 @@ The case names are:
6161
| `manual-status-500` | manual wrap with userland `dd-trace` init returning a 500 API Gateway response (`DD_TRACE_ENABLED=true`); error span tag + enhanced error metrics |
6262
| `manual-send-metrics` | manual wrap calling `sendDistributionMetric` inside and outside the handler; per-event return values |
6363
| `manual-process-input` | manual wrap with userland `dd-trace` init reading the active span; per-event return values |
64+
| `manual-callback` | manual wrap of a callback-style `(event, context, callback)` handler; pins the `promisifiedHandler` seam end to end (the migration spike broke exactly this) |
65+
| `manual-metrics-only` | `DD_TRACE_ENABLED=false` (metrics-only customers): enhanced + custom metrics still flush, no `aws.lambda` span, no trace JSON, no `dd.trace_id` log correlation |
66+
| `cjs-capture-payload` | `DD_CAPTURE_LAMBDA_PAYLOAD=true` in redirect mode; span meta gains `function.request` / `function.response` with the captured payloads |
6467
| `cjs-http-requests` | downstream HTTP calls against a hermetic mock server in redirect mode; asserts injected `x-datadog-*`/`traceparent` headers and log injection via dd-trace's http plugin |
6568
| `manual-http-requests` | same handler, manual wrap without userland dd-trace init; exercises the library's own `patchHttp` fallback (request wrapping + per-request logging + exact header set via mock echo) |
69+
| `cjs-fetch-requests` | fetch variant of `cjs-http-requests`: the global fetch (undici) is instrumented by a different dd-trace plugin than http/https; mock echo pins the injected headers on that path |
6670
| `cjs-custom-extractor` | `DD_TRACE_EXTRACTOR=extractor.extract`; asserts `_dd.parent_source: event` on the inferred span |
6771
| `cjs-proactive-init` | eager-init managed-instances RIE path with a 15 s init→invoke gap; asserts proactive-initialization markers on the raw logs |
6872

@@ -281,3 +285,16 @@ endpoints). Do not diff one suite's output against the other's snapshots.
281285
- The layer fixture installs into a plain `/opt/nodejs/node_modules`
282286
directory rather than a real published layer zip, so layer-version
283287
metadata (e.g. an exact layer ARN in tags) cannot be reproduced locally.
288+
289+
Deliberately not covered locally (each has an assigned owner — do not re-add
290+
here without closing that owner first):
291+
292+
- response streaming and `time_to_first_byte` — RIE cannot stream
293+
invocations; owned by the `serverless-e2e-tests` lambda-features suite.
294+
- direct-API and KMS/Secrets Manager metric key paths — need real AWS;
295+
unit specs plus an L3 spot-check.
296+
- aws-sdk v2/v3 client spans in the lambda context (parenting under
297+
`aws.lambda`, flush before invocation end) — need real AWS services;
298+
owned by L3.
299+
- durable-execution checkpoint extraction — owned by
300+
`serverless-e2e-tests/durable-functions`.

‎integration_tests_local/run.sh‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,14 @@
2222
# manual-status-500 | cjs | status-code-500s.handle | manual wrap; API GW 500 -> span error + enhanced error metric
2323
# manual-send-metrics | cjs | send-metrics.handle | manual wrap; sendDistributionMetric via DD_FLUSH_TO_LOG
2424
# manual-process-input | cjs | process-input.handle | manual wrap; dd-trace child spans via tracer.wrap
25+
# manual-callback | cjs | callback.handle | manual wrap; callback-style (event, context, callback) handler — the spike's proven break seam
2526
# cjs-http-requests | cjs | node_modules/datadog-lambda-js/dist/handler.handler | npm redirect; downstream HTTP header injection via dd-trace's http plugin (hermetic mock server)
2627
# manual-http-requests | cjs | http-requests-manual.handle | manual wrap; patchHttp fallback wrapping + per-request logging (hermetic mock)
28+
# cjs-fetch-requests | cjs | node_modules/datadog-lambda-js/dist/handler.handler | npm redirect; header injection on global fetch via dd-trace's undici plugin (hermetic mock)
2729
# cjs-custom-extractor | cjs | node_modules/datadog-lambda-js/dist/handler.handler | DD_TRACE_EXTRACTOR custom extractor + _dd.parent_source
2830
# cjs-proactive-init | cjs | node_modules/datadog-lambda-js/dist/handler.handler | proactive-initialization markers (raw-log assertions)
31+
# manual-metrics-only | cjs | send-metrics.handle | DD_TRACE_ENABLED=false: metrics flush, no spans, no log correlation
32+
# cjs-capture-payload | cjs | node_modules/datadog-lambda-js/dist/handler.handler | DD_CAPTURE_LAMBDA_PAYLOAD=true: function.request/response span tags
2933
#
3034
# Usage (from repo root or this directory):
3135
# ./integration_tests_local/run.sh # all runtimes, all cases
@@ -111,10 +115,14 @@ ALL_CASES=(
111115
"manual-status-500"
112116
"manual-send-metrics"
113117
"manual-process-input"
118+
"manual-callback"
114119
"cjs-http-requests"
115120
"manual-http-requests"
121+
"cjs-fetch-requests"
116122
"cjs-custom-extractor"
117123
"cjs-proactive-init"
124+
"manual-metrics-only"
125+
"cjs-capture-payload"
118126
)
119127

120128
function configure_case() {
@@ -184,6 +192,37 @@ function configure_case() {
184192
case_entry_handler="process-input.handle"
185193
case_return_mode=per-event
186194
;;
195+
manual-callback)
196+
# Callback-style (event, context, callback) handler under manual
197+
# wrap — the seam the migration spike broke (the tracePromise
198+
# wrapper replaced promisifiedHandler's call site, so callback
199+
# handlers returned null to API Gateway). handler.spec.ts pins
200+
# the unit behavior; this case pins the same path end to end
201+
# through the RIE invoke.
202+
case_image=cjs
203+
case_entry_handler="callback.handle"
204+
;;
205+
manual-metrics-only)
206+
# DD_TRACE_ENABLED=false — metrics-only customers. The golden pins
207+
# that enhanced + custom metrics still flush (via DD_FLUSH_TO_LOG)
208+
# while no aws.lambda span, no trace JSON and no dd.trace_id log
209+
# correlation appear. Same handler and per-event payloads as
210+
# manual-send-metrics; the diff between the two goldens is exactly
211+
# the tracing surface.
212+
case_image=cjs
213+
case_entry_handler="send-metrics.handle"
214+
case_extra_env=(-e DD_TRACE_ENABLED=false)
215+
case_return_mode=per-event
216+
;;
217+
cjs-capture-payload)
218+
# DD_CAPTURE_LAMBDA_PAYLOAD=true — the aws.lambda span meta gains
219+
# function.request / function.response holding the captured
220+
# payloads (event JSON is static, so the captured tags are
221+
# deterministic).
222+
case_image=cjs
223+
case_entry_handler="node_modules/datadog-lambda-js/dist/handler.handler"
224+
case_extra_env=(-e DD_LAMBDA_HANDLER=handler.handle -e DD_CAPTURE_LAMBDA_PAYLOAD=true)
225+
;;
187226
cjs-http-requests)
188227
# Redirect mode, not manual wrap: redirect mode initializes
189228
# dd-trace before the user handler loads, so the tracer's http
@@ -219,6 +258,18 @@ function configure_case() {
219258
case_needs_mock=true
220259
case_extra_env=(-e "MOCK_HTTP_URLS=http://mock-http:8080/ip-ranges-us,http://mock-http:8080/ip-ranges-eu")
221260
;;
261+
cjs-fetch-requests)
262+
# fetch/undici variant of cjs-http-requests: on Node 18+ the
263+
# global fetch is undici, instrumented by dd-trace's undici
264+
# plugin — a different injection path than the http/https plugin
265+
# that patches axios. Redirect mode, so dd-trace initializes
266+
# before the user handler loads and the plugin does the
267+
# injection; the mock echo pins the injected headers.
268+
case_image=cjs
269+
case_entry_handler="node_modules/datadog-lambda-js/dist/handler.handler"
270+
case_needs_mock=true
271+
case_extra_env=(-e DD_LAMBDA_HANDLER=fetch-requests.handle -e "MOCK_HTTP_URLS=http://mock-http:8080/ip-ranges-us,http://mock-http:8080/ip-ranges-eu")
272+
;;
222273
cjs-custom-extractor)
223274
case_image=cjs
224275
case_entry_handler="node_modules/datadog-lambda-js/dist/handler.handler"

0 commit comments

Comments
 (0)