Skip to content

test: cover callback handlers, fetch, metrics-only, payload capture in local integration harness - #823

Merged
joeyzhao2018 merged 1 commit into
mainfrom
joey/l2-coverage-pr2b
Sep 2, 2026
Merged

joeyzhao2018 merged 1 commit into
mainfrom
joey/l2-coverage-pr2b

Conversation

@joeyzhao2018

@joeyzhao2018 joeyzhao2018 commented Aug 31, 2026 •

Copy link
Copy Markdown
Collaborator

What this is

PR 2b of the dd-trace migration roadmap (Phase 0). PR 2a brought the local RIE harness to
coverage parity with the old AWS suite; this PR goes beyond the old suite, adding the
first four customer usage modes that have zero end-to-end coverage on any surface — L1
specs cover the units, but nothing pins these paths through a real invocation.

It moves no library code. Fixtures, case registrations, docs, and goldens only — all
goldens captured from the current pre-migration implementation.

Base: this branches off joey/l2-coverage-parity (PR 2a) and should land after it.

The four cases

Case What it pins Why it matters
manual-callback Callback-style (event, context, callback) handler under manual datadog() wrap, through the RIE invoke (async completion via setTimeout, so the wrapper can't mistake it for a sync return) The migration spike broke exactly this seam: the new host's tracePromise wrapper replaced promisifiedHandler's call site, and callback handlers returned null to API Gateway. handler.spec.ts pins the units; until now nothing pinned it end to end
cjs-fetch-requests Global fetch outbound in redirect mode; mock echo shows the injected x-datadog-* headers (18/18 requests carry all four) On Node 18+ fetch is undici, instrumented by dd-trace's undici plugin — a different injection path than the http/https plugin that cjs-http-requests pins via axios. Main-path pattern for Node 18+ customers
manual-metrics-only DD_TRACE_ENABLED=false: enhanced + custom metrics still flush (via DD_FLUSH_TO_LOG), and no aws.lambda span, no trace JSON, no dd.trace_id log correlation The metrics-only customer configuration — real and non-rare — had no end-to-end coverage. Return-value goldens are byte-identical to manual-send-metrics (verified), so the toggle is pinned to change only the tracing surface
cjs-capture-payload DD_CAPTURE_LAMBDA_PAYLOAD=true in redirect mode: function.request / function.response span tags holding the captured payloads Documented feature with no golden; the config-wiring phase is exactly where payload capture could drift invisibly

Docs and ledger

  • Harness README: case-table entries for the four cases, plus a "deliberately not
    covered locally"
    list in the emulation-gaps section — response streaming /
    time_to_first_byte, direct-API and KMS/Secrets metric paths, aws-sdk client spans,
    durable execution — each with its owning suite, so the scoping discussion doesn't recur.
  • migration_parity.md: the callback and payload-capture rows now point at the new
    goldens; new dd-core ownership rows for fetch/undici injection and Lambda profiling
    (DD_PROFILING_ENABLED is a documented, shipped feature that had no ledger row — the
    capability lives in dd-trace's profiler, covered by its suites plus the L3 Profiling
    row).

Verification

All four cases green in strict compare mode on the node22 and node18 legs against shared
goldens — no per-runtime overrides needed. manual-callback additionally verified on the
node26 preview leg (newest RIC + callback style was the riskiest combination). The CI
workflow runs every case per runtime leg by default, so no workflow change is needed;
expect 16 cases × 5 runtimes = 80 legs after this lands on top of PR 2a.

Remaining PR 2b items (not in this PR): kinesis/eventbridge/sns-sqs event goldens,
negative-toggle variants (DD_ENHANCED_METRICS=false / DD_LOGS_INJECTION=false), and an
authorizer encode/decode fixture.

Trying it locally

RUNTIME_PARAM=22 CASE_PARAM=manual-callback ./integration_tests_local/run.sh
RUNTIME_PARAM=18 CASE_PARAM=cjs-fetch-requests ./integration_tests_local/run.sh
RUNTIME_PARAM=22 CASE_PARAM=manual-metrics-only ./integration_tests_local/run.sh
RUNTIME_PARAM=22 CASE_PARAM=cjs-capture-payload ./integration_tests_local/run.sh

@datadog-datadog-prod-us1

This comment has been minimized.

@joeyzhao2018 joeyzhao2018 changed the title Joey/l2 coverage pr2b test: cover callback handlers, fetch, metrics-only, payload capture in local integration harness Aug 31, 2026
@joeyzhao2018
joeyzhao2018 marked this pull request as ready for review September 1, 2026 13:17
@joeyzhao2018
joeyzhao2018 requested review from a team as code owners September 1, 2026 13:17
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@joeyzhao2018
joeyzhao2018 merged commit 60b335a into main Sep 2, 2026
54 of 56 checks passed
@joeyzhao2018
joeyzhao2018 deleted the joey/l2-coverage-pr2b branch September 2, 2026 17:14
joeyzhao2018 added a commit that referenced this pull request Sep 17, 2026
… serverless-e2e-tests)

The integration_tests/ AWS suite is retired. Its coverage now lives in two
places, each stronger at what it owns:

- Behavior: integration_tests_local/ (docker/RIE, GitHub Actions, per PR) is
  now the behavior oracle — hermetic, contributor-runnable, and strictly
  stronger than the AWS suite at behavior (its goldens preserve span
  meta/metrics keys the AWS normalization stripped wholesale). After #821 and
  #823 it covers every variant the AWS suite pinned, including callback-style
  handlers, fetch, metrics-only mode, payload capture, and the manual-wrap
  no-init path.

- Real-AWS-only signals: the four variants whose value IS the platform —
  esm (published layer zip artifact + ESM loader), sync-metrics (direct-API
  sink to the real intake), http-requests (_X_AMZN_TRACE_ID pass-through +
  real egress), status-code-500s (real API Gateway) — move to a dedicated
  integration-tests-residual suite in serverless-e2e-tests, which deploys
  against published/candidate layer ARNs and keeps the same snapshots and
  byte-identical normalization pipeline.

Deleted here: serverless.yml, the six top-level handlers, all 450 snapshots,
and scripts/run_integration_tests.sh (459 files). Kept: input_events/,
parse-json.js, and container/ — the local suite references the first two and
uses container/ as fixture; a new integration_tests/README.md documents this.

CI: the GitLab `integration test` job and its sign/publish `needs` entries
are removed (build-layer keeps CI_ENABLE_CONTAINER_IMAGE_BUILDS, which it
uses to docker-build the layer). CONTRIBUTING.md now points contributors at
the local suite.

Release-gate note: the per-pipeline AWS gate on sign/publish goes away. The
candidate-layer gate moves to the residual suite in serverless-e2e-tests —
manual until the e2e release-gate automation lands (migration roadmap), so
release runbooks should trigger it until then.
joeyzhao2018 added a commit that referenced this pull request Sep 17, 2026
… serverless-e2e-tests)

The integration_tests/ AWS suite is retired. Its coverage now lives in two
places, each stronger at what it owns:

- Behavior: integration_tests_local/ (docker/RIE, GitHub Actions, per PR) is
  now the behavior oracle — hermetic, contributor-runnable, and strictly
  stronger than the AWS suite at behavior (its goldens preserve span
  meta/metrics keys the AWS normalization stripped wholesale). After #821 and
  #823 it covers every variant the AWS suite pinned, including callback-style
  handlers, fetch, metrics-only mode, payload capture, and the manual-wrap
  no-init path.

- Real-AWS-only signals: the four variants whose value IS the platform —
  esm (published layer zip artifact + ESM loader), sync-metrics (direct-API
  sink to the real intake), http-requests (_X_AMZN_TRACE_ID pass-through +
  real egress), status-code-500s (real API Gateway) — move to a dedicated
  integration-tests-residual suite in serverless-e2e-tests, which deploys
  against published/candidate layer ARNs and keeps the same snapshots and
  byte-identical normalization pipeline.

Deleted here: serverless.yml, the six top-level handlers, all 450 snapshots,
and scripts/run_integration_tests.sh (459 files). Kept: input_events/,
parse-json.js, and container/ — the local suite references the first two and
uses container/ as fixture; a new integration_tests/README.md documents this.

CI: the GitLab `integration test` job and its sign/publish `needs` entries
are removed (build-layer keeps CI_ENABLE_CONTAINER_IMAGE_BUILDS, which it
uses to docker-build the layer). CONTRIBUTING.md now points contributors at
the local suite.

Release-gate note: the per-pipeline AWS gate on sign/publish goes away. The
candidate-layer gate moves to the residual suite in serverless-e2e-tests —
manual until the e2e release-gate automation lands (migration roadmap), so
release runbooks should trigger it until then.
joeyzhao2018 added a commit that referenced this pull request Sep 17, 2026
… serverless-e2e-tests)

The integration_tests/ AWS suite is retired. Its coverage now lives in two
places, each stronger at what it owns:

- Behavior: integration_tests_local/ (docker/RIE, GitHub Actions, per PR) is
  now the behavior oracle — hermetic, contributor-runnable, and strictly
  stronger than the AWS suite at behavior (its goldens preserve span
  meta/metrics keys the AWS normalization stripped wholesale). After #821 and
  #823 it covers every variant the AWS suite pinned, including callback-style
  handlers, fetch, metrics-only mode, payload capture, and the manual-wrap
  no-init path.

- Real-AWS-only signals: the four variants whose value IS the platform —
  esm (published layer zip artifact + ESM loader), sync-metrics (direct-API
  sink to the real intake), http-requests (_X_AMZN_TRACE_ID pass-through +
  real egress), status-code-500s (real API Gateway) — move to a dedicated
  integration-tests-residual suite in serverless-e2e-tests, which deploys
  against published/candidate layer ARNs and keeps the same snapshots and
  byte-identical normalization pipeline.

Deleted here: serverless.yml, the six top-level handlers, all 450 snapshots,
and scripts/run_integration_tests.sh (459 files). Kept: input_events/,
parse-json.js, and container/ — the local suite references the first two and
uses container/ as fixture; a new integration_tests/README.md documents this.

CI: the GitLab `integration test` job and its sign/publish `needs` entries
are removed (build-layer keeps CI_ENABLE_CONTAINER_IMAGE_BUILDS, which it
uses to docker-build the layer). CONTRIBUTING.md now points contributors at
the local suite.

Release-gate note: the per-pipeline AWS gate on sign/publish goes away. The
candidate-layer gate moves to the residual suite in serverless-e2e-tests —
manual until the e2e release-gate automation lands (migration roadmap), so
release runbooks should trigger it until then.
joeyzhao2018 added a commit that referenced this pull request Sep 18, 2026
…e e2e and the RIE based tests (#826)

* test: retire the AWS-based integration suite (moved to local docker + serverless-e2e-tests)

The integration_tests/ AWS suite is retired. Its coverage now lives in two
places, each stronger at what it owns:

- Behavior: integration_tests_local/ (docker/RIE, GitHub Actions, per PR) is
  now the behavior oracle — hermetic, contributor-runnable, and strictly
  stronger than the AWS suite at behavior (its goldens preserve span
  meta/metrics keys the AWS normalization stripped wholesale). After #821 and
  #823 it covers every variant the AWS suite pinned, including callback-style
  handlers, fetch, metrics-only mode, payload capture, and the manual-wrap
  no-init path.

- Real-AWS-only signals: the four variants whose value IS the platform —
  esm (published layer zip artifact + ESM loader), sync-metrics (direct-API
  sink to the real intake), http-requests (_X_AMZN_TRACE_ID pass-through +
  real egress), status-code-500s (real API Gateway) — move to a dedicated
  integration-tests-residual suite in serverless-e2e-tests, which deploys
  against published/candidate layer ARNs and keeps the same snapshots and
  byte-identical normalization pipeline.

Deleted here: serverless.yml, the six top-level handlers, all 450 snapshots,
and scripts/run_integration_tests.sh (459 files). Kept: input_events/,
parse-json.js, and container/ — the local suite references the first two and
uses container/ as fixture; a new integration_tests/README.md documents this.

CI: the GitLab `integration test` job and its sign/publish `needs` entries
are removed (build-layer keeps CI_ENABLE_CONTAINER_IMAGE_BUILDS, which it
uses to docker-build the layer). CONTRIBUTING.md now points contributors at
the local suite.

Release-gate note: the per-pipeline AWS gate on sign/publish goes away. The
candidate-layer gate moves to the residual suite in serverless-e2e-tests —
manual until the e2e release-gate automation lands (migration roadmap), so
release runbooks should trigger it until then.

* merge main

* docs: clarify AWS integration test deprecation

* ci: retry pipeline

* test: pin RIE tracer per runtime

* ci: retrigger pipeline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants