test: expand local integration harness to old-suite coverage parity (#821)

* test: expand local integration harness to old-suite coverage parity

PR 2a of the dd-trace migration roadmap: freeze goldens at the coverage
level of the AWS-based suite, not just the two container variants.

Cases (was 2, now 11): layer-cjs/layer-esm (new /opt layer fixture via
prepare-layer.js, lockfile-pinned deps), manual-throw-error /
manual-status-500 / manual-send-metrics / manual-process-input (AWS
suite's manual-wrap handlers ported into the cjs fixture),
cjs-http-requests (hermetic mock server on a per-run docker network),
cjs-custom-extractor (DD_TRACE_EXTRACTOR, asserts _dd.parent_source),
cjs-proactive-init (managed-instances eager init + 15s gap, raw-log
marker assertions).

Harness: run.sh case engine (CASE_PARAM; VARIANT_PARAM and
SIMULATE_PROACTIVE_INIT kept as aliases), per-case return-value modes
(default/case/per-event) with most-specific-first golden resolution;
assert-then-collapse extended to the dd_lambda_layer node major;
normalize.sh rules for ISO8601-T RIC timestamps, compact JSON trace
headers, requestId forms, TimeoutOverflowWarning values, node pids, and
the managed-instances supervisor pid.

Per-runtime override goldens pin genuine divergences: Node 24+/26 error
stack frames, Node 18 dns/net plugin spans, per-major
TimeoutOverflowWarning emission in the proactive case.

Verified: all 11 cases pass strict compare on nodejs 18/20/22/24/26.

* fix: normalize Node's racy once-per-process trace-warnings hint

Node appends "(Use `node --trace-warnings ...` to show where the warning
was created)" to the first warning a process emits. Under the
managed-instances path, which warning is first per process is racy, so the
hint's distribution across TimeoutOverflowWarning records differs between
runs — and between arm64 local and amd64 CI, which is what failed the
node22/node24 legs on the PR.

Drop the hint in both its forms (JSON-embedded `\n(...)` suffix and
standalone line) in normalize.sh and re-capture the four proactive-init
goldens that contain it. Verified: strict compare passes for
cjs-proactive-init on all five runtimes.

* add manual datadog(handle) case where tracer isn't initialized.

* local integration tests: stay runnable on dd-trace v6 trees

The pack step ran `yarn install --frozen-lockfile` directly. On a tree that
pins dd-trace v6 (engines node >=22) with a host Node older than 22 — e.g.
this repo's GitHub Actions runner image, which sets up Node 18.12 — that
install fails with "The engine \"node\" is incompatible with this module",
so every local integration test leg on a v6 PR fails before any container
is built.

Two changes, both no-ops on the current v5 tree:

- Pack step: when scripts/install_deps.sh exists (v6 world), install
  through it with TARGET_NODE_MAJOR=$RUNTIME_PARAM (default 22) so the v5
  fallback line is installed for older runtimes and --ignore-engines
  covers v6 on older hosts. The script restores package.json/yarn.lock on
  exit, so the worktree stays clean. Without it, fall back to the previous
  plain yarn install.
- Container builds: pass a DD_TRACE_VERSION build-arg resolved through
  scripts/dd_trace_versions.sh (v5 line for node <22) so the fixture image
  installs a tracer its runtime can actually run. Current fixture
  Dockerfiles declare no such ARG, so docker only warns; v6-world fixtures
  consume it via npm pkg set.

Verified:
- v5 tree: RUNTIME_PARAM=22 CASE_PARAM=container-cjs passes end to end.
- v6 tree (onzia/moreddtrace6 + this harness): node18 leg installs
  dd-trace 5.124.0 and node22 leg installs 6.12.0 on a Node 18 host (the
  original failure scenario), containers build and run; the only remaining
  diffs are the expected tracer-version snapshot drift (5.118 -> 5.124/6.x
  drop empty "links": [] and populate dd_trace meta), which the v6 PR owns
  regenerating, same as it already did for the AWS snapshots.

* normalize dns.addresses values

Author: joeyzhao2018

.github/workflows/integration_tests_local.yml

@@ -4,11 +4,12 @@ name: local integration tests
 # against the real AWS Lambda base images + the Runtime Interface Emulator
 # (RIE). No AWS account or credentials required.
 #
-# One job per Node major version; each job builds and tests both container
-# variants (cjs, esm), which share the same ~1 GB base image, so each job
-# pulls it exactly once. If public.ecr.aws anonymous pull limits ever bite,
-# the fix is to mirror the base images to GHCR (same pattern dd-trace-js
-# uses in its mirror-image.yml workflow).
+# One job per Node major version; each job builds the container/layer fixture
+# images and runs every case (default: all of them) against that runtime.
+# The fixtures share the same ~1 GB base image, so each job pulls it exactly
+# once. If public.ecr.aws anonymous pull limits ever bite, the fix is to
+# mirror the base images to GHCR (same pattern dd-trace-js uses in its
+# mirror-image.yml workflow).
 
 on:
   pull_request:

integration_tests/container/cjs/Dockerfile

@@ -1,7 +1,11 @@
 ARG NODE_VERSION=22
 FROM public.ecr.aws/lambda/nodejs:${NODE_VERSION}
 
-COPY package.json handler.js ${LAMBDA_TASK_ROOT}/
+# handler.js is the default redirect-mode fixture (DD_LAMBDA_HANDLER=handler.handle).
+# The remaining *.js files are the manual-wrap fixture handlers ported from the
+# AWS suite (throw-error, status-code-500s, send-metrics, process-input,
+# http-requests) plus extractor.js for the DD_TRACE_EXTRACTOR case.
+COPY package.json *.js ${LAMBDA_TASK_ROOT}/
 COPY datadog-lambda-js-local.tgz /tmp/datadog-lambda-js-local.tgz
 RUN cd ${LAMBDA_TASK_ROOT} \
     && npm install --omit=dev \

integration_tests/container/cjs/extractor.js

@@ -0,0 +1,15 @@
+// Custom trace-context extractor fixture, loaded via
+// DD_TRACE_EXTRACTOR=extractor.extract (handler-style "module.function"
+// string, resolved by the shim's user-function loader).
+//
+// Returns a fixed context on every event: the extracted parent and the
+// `_dd.parent_source` span tag must show up in the trace, which the golden
+// asserts deterministically (ids are normalized to XXXX; the tag is not).
+module.exports.extract = async (event, context) => {
+  return {
+    traceId: "1234567890123456789",
+    parentId: "9876543210987654321",
+    sampleMode: 1,
+    source: "event",
+  };
+};

integration_tests/container/cjs/http-requests-manual.js

@@ -0,0 +1,18 @@
+// Manual-wrap companion to http-requests.js, run as the manual-http-requests
+// case — the exact shape of the AWS suite's http-requests function
+// (manual datadog() wrap, no userland dd-trace init).
+//
+// With no tracer initialized, TraceListener falls back to the library's own
+// patchHttp (src/trace/patch-http.ts), which wraps http/https and logs
+// "HTTP GET <url> TraceHeaders: [...]" per request. On real Lambda the
+// injected headers come from the platform's pass-through _X_AMZN_TRACE_ID;
+// under RIE that variable cannot be emulated (the RIC owns it), so the
+// golden pins the empty-context form (TraceHeaders: []) plus the mock echo
+// of the exact header set sent. The path under test — patchHttp wiring —
+// fails this golden the moment it stops wrapping or logging; the
+// context-dependent header values are pinned by the patch-http unit tests
+// and the AWS suite.
+const { datadog } = require("datadog-lambda-js");
+const { handle } = require("./http-requests");
+
+module.exports.handle = datadog(handle);

integration_tests/container/cjs/http-requests.js

@@ -0,0 +1,40 @@
+// Ported from integration_tests/http-requests.js with one seam adaptation:
+// the AWS suite hits the real ip-ranges.datadoghq.com/.eu endpoints; the
+// local harness is hermetic, so URLs come from MOCK_HTTP_URLS (set by run.sh
+// to the mock-http container on the case's docker network). The default keeps
+// this file runnable outside the harness.
+//
+// This entry point is UNWRAPPED and runs through the npm redirect entry
+// (dist/handler.handler) in the cjs-http-requests case. Redirect mode
+// initializes dd-trace first, so the tracer's http plugin patches axios and
+// the mock echo below shows the injected x-datadog-* headers in the golden.
+// The manual-wrap variant of the AWS suite's function — which exercises the
+// library's own patchHttp fallback instead — lives in http-requests-manual.js
+// and runs as the manual-http-requests case.
+//
+// The mock echoes the request headers it received, and this handler logs them,
+// so both goldens show the injected downstream trace context
+// (x-datadog-trace-id / x-datadog-parent-id, normalized to XXXX).
+const { sendDistributionMetric } = require("datadog-lambda-js");
+const axios = require("axios");
+
+const urls = (process.env.MOCK_HTTP_URLS ||
+  "https://ip-ranges.datadoghq.com,https://ip-ranges.datadoghq.eu"
+).split(",");
+
+async function handle(event, context) {
+  const responsePayload = { message: "hello, dog!" };
+
+  sendDistributionMetric("serverless.integration_test.execution", 1, "function:http-request");
+
+  for (let index = 0; index < urls.length; index++) {
+    const response = await axios({ url: urls[index], method: "get" });
+    console.log(`mock-http saw headers for ${urls[index]}: ${JSON.stringify(response.data.headers)}`);
+  }
+
+  console.log(`Snapshot test http requests successfully made to URLs: ${urls}`);
+
+  return responsePayload;
+}
+
+module.exports.handle = handle;

integration_tests/container/cjs/package.json

@@ -3,6 +3,7 @@
   "version": "1.0.0",
   "private": true,
   "dependencies": {
+    "axios": "1.18.1",
     "dd-trace": "5.105.0"
   }
 }

integration_tests/container/cjs/process-input.js

@@ -0,0 +1,91 @@
+const tracer = require("dd-trace").init();
+
+const { datadog, sendDistributionMetric } = require("datadog-lambda-js");
+
+// Minimal example of tracing to reduce flakiness
+async function handle(event, context) {
+  const span = tracer.scope().active();
+
+  const responsePayload = { message: "hello, dog!" };
+
+  sendDistributionMetric("serverless.integration_test.execution", 1, "function:process-input-traced");
+
+  const { recordIds, eventType } = getRecordIds(event);
+
+  if (recordIds) {
+    responsePayload.recordIds = recordIds;
+  }
+
+  if (eventType) {
+    responsePayload.eventType = eventType;
+  } else {
+    const requestId = getAPIGatewayRequestId(event);
+
+    if (requestId) {
+      responsePayload.eventType = "APIGateway";
+
+      if (span) {
+        span.setTag("api_gateway_request_id", requestId);
+      }
+    }
+  }
+
+  if (span) {
+    span.setTag("event_type", responsePayload.eventType);
+  }
+
+  return responsePayload;
+}
+
+const getRecordIds = tracer.wrap("getRecordIds", (event) => {
+  const recordIds = [];
+  let eventType = null;
+  if (event.Records) {
+    event.Records.forEach((record) => {
+      if (record.messageId) {
+        eventType = "SQS";
+        recordIds.push(record.messageId);
+      }
+      if (record.Sns) {
+        eventType = "SNS";
+        recordIds.push(record.Sns.MessageId);
+      }
+      if (record.s3) {
+        eventType = "S3";
+        const bucketName = record.s3.bucket.name;
+        const objectKey = record.s3.object.key;
+        recordIds.push(`${bucketName}/${objectKey}`);
+      }
+    });
+  }
+
+  if (eventType) {
+    const span = tracer.scope().active();
+
+    if (span) {
+      span.setTag("record_event_type", eventType);
+      span.setTag("record_ids", recordIds.join());
+    }
+  }
+
+  return { recordIds, eventType };
+});
+
+const getAPIGatewayRequestId = tracer.wrap("getAPIGatewayRequestId", (event) => {
+  let requestId;
+
+  if (event.requestContext) {
+    requestId = event.requestContext.requestId;
+
+    const span = tracer.scope().active();
+
+    if (span) {
+      span.setTag("api_gateway_request_id", requestId);
+      span.setTag("event_type", "APIGateway");
+    }
+  }
+
+  return requestId;
+});
+
+module.exports.handle = datadog(handle);

integration_tests/container/cjs/send-metrics.js

@@ -0,0 +1,57 @@
+const { datadog, sendDistributionMetric } = require("datadog-lambda-js");
+
+sendDistributionMetric(
+  "serverless.integration_test.outside_handler",
+  1,
+  "tagkey:tagvalue",
+  `eventsource:outside_handler`,
+);
+
+async function handle(event, context) {
+  const responsePayload = { message: "hello, dog!" };
+
+  if (event.requestContext) {
+    responsePayload.eventType = "APIGateway";
+    responsePayload.requestId = event.requestContext.requestId;
+  }
+
+  if (event.Records) {
+    responsePayload.recordIds = [];
+
+    event.Records.forEach((record) => {
+      if (record.messageId) {
+        responsePayload.eventType = "SQS";
+        responsePayload.recordIds.push(record.messageId);
+        sendDistributionMetric(
+          "serverless.integration_test.records_processed",
+          1,
+          "tagkey:tagvalue",
+          `eventsource:${responsePayload.eventType}`,
+        );
+      }
+      if (record.Sns) {
+        responsePayload.eventType = "SNS";
+        responsePayload.recordIds.push(record.Sns.MessageId);
+        sendDistributionMetric(
+          "serverless.integration_test.records_processed",
+          1,
+          "tagkey:tagvalue",
+          `eventsource:${responsePayload.eventType}`,
+        );
+      }
+    });
+  }
+
+  sendDistributionMetric(
+    "serverless.integration_test.execution",
+    1,
+    "tagkey:tagvalue",
+    `eventsource:${responsePayload.eventType}`,
+  );
+
+  console.log(`Processed ${responsePayload.eventType} request`);
+
+  return responsePayload;
+}
+
+module.exports.handle = datadog(handle);

integration_tests/container/cjs/status-code-500s.js

@@ -0,0 +1,15 @@
+// Ported as-is from integration_tests/status-code-500s.js (the AWS suite's
+// API Gateway 500 case): returns statusCode 500 so the span gets the error
+// tag and the aws.lambda.enhanced.errors metric is emitted.
+const tracer = require("dd-trace").init();
+const { datadog } = require("datadog-lambda-js");
+
+async function handle(event, context) {
+  const span = tracer.scope().active();
+  return {
+    "statusCode": 500,
+    "body": {}
+  };
+}
+
+module.exports.handle = datadog(handle);

integration_tests/container/cjs/throw-error.js

@@ -0,0 +1,10 @@
+// Ported as-is from integration_tests/throw-error-traced.js (the AWS suite's
+// manual-wrap error case): a handler that always throws, wrapped with
+// datadog() manually. Exercises thrown-error -> span error + rethrow through
+// the RIE invoke path.
+const { datadog } = require("datadog-lambda-js");
+
+async function handle(event, context) {
+  throw new Error("Hello");
+}
+module.exports.handle = datadog(handle);