Skip to content

Commit dabb8e7

Browse files
joeyzhao2018rithikanarayanclaude
authored
test: add local Docker/RIE integration test harness (Level 2 tests) (#804)
* make integration tests run local instead of using real aws service * test(local): keep cold_start asserted in local integration snapshots Remove the cold_start value normalization from the local harness pipeline. The cold->warm transition (invoke #1 cold_start:true, invokes #2..N false) is deliberate coverage and is deterministic locally, since proactive initialization cannot happen unless SIMULATE_PROACTIVE_INIT=true. Snapshots regenerated with real cold_start values. proactive_initialization markers and the '(init: N ms)' END suffix remain stripped: they reflect platform scheduling, not code behavior. The real-AWS run_integration_tests.sh is intentionally untouched; proactive-init cold_start flakes there are handled by rerun. * ci: run local integration tests in GitHub Actions Adds a workflow that runs the RIE-based harness from integration_tests_local/ on every PR and main push: one job per Node major version (18/20/22/24), each building and testing both container variants against the real AWS Lambda base images. No AWS credentials needed. Uses linux/amd64 images on the x86_64 runners. * test(local): drop 'container-' prefix from snapshot file names Snapshot files are now named <variant>_node<runtime>[...] (e.g. esm_node18.log) instead of container-<variant>_node<runtime>[...]. The docker image tags and function names keep the container- prefix; only the snapshot paths changed. * Sign the datadog-lambda-js ci image (#809) * Pin serverless version (#810) * Remove use of Github secrets (#811) * Add Node26 runtime (#808) * Add Node26 runtime * update snapshots * fix integration test path * node version compatibilty * filtering snapshots * cleanup * add migration_parity.md doc * update * add node26 * Harden local RIE integration harness (Phase 0 PR 1) Prepare the docker/RIE golden harness to serve as the compatibility oracle for the dd-trace-js migration, before any migrated code runs against it. Oracle correctness: - normalize.sh: `run_id_filter` defaulted to 's/$/^/', which is not a no-op -- it appended a literal '^' to every output line, and all eight committed log goldens carried one. Setting RUN_ID would have flipped the filter and broken them all at once. Default is now a genuine pass-through, and the goldens are updated by that byte-provable transform only (strip the trailing '^', strip RIE's trailing REPORT tab); no golden was regenerated by running code. - normalize.sh: stop wiping span `meta`/`metrics` keys -- their stable keys and values are part of the compatibility contract. (The filter was already dead: the goldens are pretty-printed and the pattern could not cross newlines.) - Trailing-blank stripping is scoped to REPORT lines, so application output remains an exact-whitespace oracle. - run.sh: drop `diff -w` from the sorted log comparison. It hid whitespace-only drift, e.g. a migrated emitter serializing '{"a":1}' where today it emits '{"a": 1}'. The tab it existed for is now removed at the source. Reliability: - Pin the RIE binary to v1.36 with a per-platform SHA-256, re-hashed on every run so a stale cached download cannot survive. - Fail on missing snapshots unless UPDATE_SNAPSHOTS=true; validate RUNTIME_PARAM/VARIANT_PARAM/UPDATE_SNAPSHOTS; reject argv, which was silently ignored. - Capture the HTTP status alongside the body: `curl -s` exits 0 on a 5xx, so update mode could have recorded error pages as goldens. - Handle `diff` exit codes 0/1/>1 distinctly; a diff *error* previously read as success. - Replace fixed sleeps with polling: a TCP readiness probe (never a POST, which would consume the cold-start invocation) and a wait for both REPORT and INVOKE RTDONE per event, since RIE can emit RTDONE after REPORT. - Guard cleanup() against the blank array slots left by "${array[@]/$cid}". Node 26: - Add the leg to the harness and CI matrix. Node 26 ships to customers but has zero L2 coverage. - `public.ecr.aws/lambda/nodejs:26` does not exist yet (preview only), so pin the multi-arch tag 26-preview.2026.08.21.22 through a runtime->image-tag indirection applied only at the --build-arg; $node_version also feeds image tags, container names and snapshot paths, which must keep the bare major. - Node 26 is excluded from the default runtime list and runs in update mode in CI until PR 2 commits its snapshots. CI asserts it produces exactly 20 artifacts and that no committed snapshot is ever mutated. - Record the GA re-capture trigger: when AWS publishes the bare image, the Node 26 goldens must be re-captured from the pinned pre-migration ref. Node 18 and 20 legs are retained deliberately (C18); v5.x is the only dd-trace line serving those runtimes. Verified: all eight strict legs (18/20/22/24 x cjs/esm) pass byte-exact; the node26 update-mode leg produces exactly 20 artifacts; negative tests for missing snapshots and whitespace-only drift both fail as intended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: share local integration goldens across runtimes and events The local harness committed one golden per (variant, runtime, event): 72 return values and 8 logs. All 72 return values were byte-identical, and the 8 logs held only one genuinely runtime-specific line each. 80 files carried 3 distinct expectations, and 80 files had to be re-read whenever one of them legitimately changed. Collapse them to 3 shared goldens covering all five runtimes and both variants: snapshots/return_values/default.json snapshots/logs/cjs.log snapshots/logs/esm.log Two changes make sharing safe rather than lossy: - Assert `runtime:nodejsNN.x` on every invocation with the major actually under test, then collapse it to `nodejsXX.x`. Without the assertion, sharing the golden would silently stop checking that the library reports the runtime it is running on. - Drop the runtime major from AWS_LAMBDA_FUNCTION_NAME. It propagates into service, resource, resource_names, functionname, function_arn, _dd.base_service and _dd.tags.process, so embedding the runtime there made every runtime's golden differ in ~100 lines of pure fixture naming, which buried the one line that was actually runtime-specific. Divergence is expressed by adding an override file, never by loosening a comparison: snapshots/logs/${variant}_node${major}.log snapshots/return_values/${variant}_node${major}_${event}.json An override wins for its leg alone, so a real behavioral difference appears as a new file in review, where widening a normalization filter to absorb it would appear as nothing. In update mode a leg that disagrees with an existing shared golden now fails instead of overwriting it; otherwise the last runtime to run would define the expectation for all of them. That agreement check uses the same sort-insensitive comparison as strict mode, since RIE's platform logging races application output. Node 26 consequently needs no snapshots of its own: its normalized output matches the shared goldens. That removes the update-mode CI override and the node26 baseline assertion, making all five workflow legs strict. The consolidation is auditable rather than a blind re-capture: reverse- substituting the runtime major and function name into the 3 shared files reproduces all 72 original return-value goldens byte-for-byte and all 8 original per-runtime logs with an identical set of lines. The only residual differences are the ordering of RIE's own platform lines (INVOKE RTDONE, START/END/REPORT), which is already compared order-insensitively. Also fixes a latent normalization hole this exposed: function_arn is normalized by a pattern whose character class excludes `_`, so names ending in `_node18` halted the match and the goldens recorded a partially normalized `"function_arn":"XXXX_node18"`. With the suffix gone the ARN normalizes completely. Verified with a full strict run: 5 runtimes x 2 variants, 110 assertions, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: unbreak SIMULATE_PROACTIVE_INIT under the RTDONE readiness poll The readiness poll waits for both a REPORT and an INVOKE RTDONE per invocation. RIE's managed-instances path — the one SIMULATE_PROACTIVE_INIT switches into to get eager init — emits REPORT but never RTDONE, so the poll could never be satisfied and the mode timed out at 9/0 instead of producing a usable diff. Verified directly against both RIE paths with a single invocation: default RIE REPORT=1 INVOKE_RTDONE=1 managed-instances REPORT=1 INVOKE_RTDONE=0 Require RTDONE only on the default path. The mode now runs to completion and surfaces exactly what it is meant to: cold_start flips to false on the first invocation, plus managed-instances platform noise (structured bootstrap logs and a different emulated region in ARNs). Also correct the proactive_initialization row in migration_parity.md, which claimed RIE cannot produce a proactively-initialized sandbox. It can: raw un-normalized logs from a managed-instances run with a 15s init->invoke gap contain "proactive_initialization":1, proactive_initialization:true and cold_start:false. The feature is unprotected because normalization drops those markers and the goldens come from immediate-invoke runs, not because the sandbox state is unreachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: state plainly that SIMULATE_PROACTIVE_INIT is not a gate The section documented the mechanism in detail but never said what its status is, so a reader could reasonably assume it is part of the suite. It is not set in CI, not set in the default run, and nothing gates on it; a run with it is expected to differ from the goldens rather than match them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Rithika Narayan <93233069+rithikanarayan@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 2d3f33d commit dabb8e7

10 files changed

Lines changed: 2795 additions & 1 deletion

File tree

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: local integration tests
2+
3+
# Runs the docker-only integration test harness in integration_tests_local/
4+
# against the real AWS Lambda base images + the Runtime Interface Emulator
5+
# (RIE). No AWS account or credentials required.
6+
#
7+
# One job per Node major version; each job builds and tests both container
8+
# variants (cjs, esm), which share the same ~1 GB base image, so each job
9+
# pulls it exactly once. If public.ecr.aws anonymous pull limits ever bite,
10+
# the fix is to mirror the base images to GHCR (same pattern dd-trace-js
11+
# uses in its mirror-image.yml workflow).
12+
13+
on:
14+
pull_request:
15+
push:
16+
branches:
17+
- "main"
18+
workflow_dispatch:
19+
20+
jobs:
21+
container:
22+
runs-on: ubuntu-latest
23+
24+
strategy:
25+
fail-fast: false
26+
matrix:
27+
node_major: ["18", "20", "22", "24", "26"]
28+
29+
steps:
30+
- name: Checkout
31+
uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
32+
33+
- name: Set up Node 18.12
34+
uses: actions/setup-node@3235b876344d2a9aa001b8d1453c930bba69e610 # v3.9.1
35+
with:
36+
node-version: 18.12
37+
38+
- name: Run local container integration tests (nodejs${{ matrix.node_major }}.x)
39+
run: PLATFORM=linux/amd64 RUNTIME_PARAM=${{ matrix.node_major }} ./integration_tests_local/run.sh
40+
41+
# Every leg is a strict comparison, so a run must leave the snapshots
42+
# untouched. Writing here would mean the harness recorded its own output
43+
# as the expectation instead of checking against it.
44+
- name: Assert no snapshot was written
45+
if: always()
46+
run: |
47+
if ! git diff --exit-code --stat integration_tests_local/snapshots; then
48+
echo "FAILURE: this run modified committed snapshots" >&2
49+
exit 1
50+
fi
51+
new_files=$(git ls-files --others --exclude-standard integration_tests_local/snapshots)
52+
if [ -n "$new_files" ]; then
53+
echo "FAILURE: strict leg created snapshots instead of comparing:" >&2
54+
echo "$new_files" >&2
55+
exit 1
56+
fi

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,6 @@ package-lock.json
1717

1818
# Local datadog-lambda-js tarballs produced for container integration tests
1919
integration_tests/container/*/datadog-lambda-js-local.tgz
20+
21+
# Local integration test harness binaries
22+
integration_tests_local/bin/

‎integration_tests_local/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
bin/

‎integration_tests_local/README.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Local integration tests (docker + AWS RIE, no AWS account required)
2+
3+
This directory contains a **local** integration test harness for
4+
datadog-lambda-js. It runs the container-image handler variants
5+
(`integration_tests/container/cjs` and `integration_tests/container/esm`)
6+
inside Docker against the
7+
[AWS Lambda Runtime Interface Emulator (RIE)](https://github.com/aws/aws-lambda-runtime-interface-emulator),
8+
invokes them with the same input events as the AWS-based suite, captures
9+
logs from `docker logs`, normalizes them with `./normalize.sh` — the AWS
10+
suite's filter chain plus documented RIE-specific handling, not an identical
11+
copy — and diffs them against **local** snapshots in `./snapshots/`.
12+
13+
Nothing here touches AWS, and nothing here touches
14+
`integration_tests/snapshots/` (the AWS suite's snapshots).
15+
16+
## Prerequisites
17+
18+
- Docker (tested with colima on macOS arm64)
19+
- `node`, `yarn`, `perl`, `sed`, `curl` (all already used by the AWS suite)
20+
- Network access for the first run (pulls `public.ecr.aws/lambda/nodejs:*`
21+
base images, ~1 GB each, and downloads the pinned RIE binary into `./bin/`)
22+
23+
## Running
24+
25+
```bash
26+
# Everything: nodejs 18/20/22/24/26 x {cjs, esm}
27+
./integration_tests_local/run.sh
28+
29+
# One runtime / one variant
30+
RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh
31+
32+
# (Re)generate local snapshots
33+
UPDATE_SNAPSHOTS=true ./integration_tests_local/run.sh
34+
35+
# Skip repacking the library (reuse container/*/datadog-lambda-js-local.tgz)
36+
SKIP_PACK=true RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh
37+
38+
# Force amd64 images instead of arm64
39+
PLATFORM=linux/amd64 ./integration_tests_local/run.sh
40+
```
41+
42+
Unless `SKIP_PACK=true` is set, each run repacks the library under test
43+
(`yarn install --frozen-lockfile && yarn build && npm pack`) into
44+
`integration_tests/container/{cjs,esm}/datadog-lambda-js-local.tgz`, exactly
45+
like `scripts/run_integration_tests.sh` does, so the containers always test
46+
the working tree.
47+
48+
The harness takes no flags or positional arguments — configuration is
49+
environment-only, and an unexpected argument is rejected rather than ignored.
50+
51+
Without `UPDATE_SNAPSHOTS=true`, every expected return-value and log snapshot
52+
must already exist. A missing snapshot fails the run and is never created
53+
implicitly. Update mode is the only path that creates or overwrites snapshots.
54+
55+
## Snapshot layout: shared goldens with per-case overrides
56+
57+
There are three committed snapshots, not one per case:
58+
59+
```
60+
snapshots/return_values/default.json # every event, every runtime, every variant
61+
snapshots/logs/cjs.log # nodejs 18/20/22/24/26, cjs
62+
snapshots/logs/esm.log # nodejs 18/20/22/24/26, esm
63+
```
64+
65+
All 9 input events return the same fixture response, and the normalized logs
66+
are identical across all five runtimes, so a per-case file per leg would be 80
67+
copies of 3 distinct expectations — and 80 files to review when one of them
68+
legitimately changes.
69+
70+
Two things make the sharing safe rather than lossy:
71+
72+
- **The runtime is asserted before it is collapsed.** `runtime:nodejsNN.x` is
73+
the only genuinely runtime-specific line in the log. `run.sh` checks that it
74+
appears on every invocation with the major actually under test, and only then
75+
rewrites it to `nodejsXX.x`. Sharing the golden therefore does not stop the
76+
suite from checking that the library reports the runtime it is running on.
77+
- **`AWS_LAMBDA_FUNCTION_NAME` carries no runtime major.** It propagates into
78+
`service`, `resource`, `resource_names`, `functionname`, `function_arn`,
79+
`_dd.base_service` and `_dd.tags.process`, so embedding the runtime there
80+
made every runtime's golden differ in ~100 lines of pure fixture naming,
81+
burying the one line that was actually runtime-specific.
82+
83+
Divergence is expressed by **adding a file**, never by loosening a comparison:
84+
85+
```
86+
snapshots/logs/${variant}_node${major}.log # overrides the shared log
87+
snapshots/return_values/${variant}_node${major}_${event}.json # overrides the shared return value
88+
```
89+
90+
When an override is present it wins for that leg alone. This keeps a real
91+
behavioral difference visible in review, whereas widening a normalization
92+
filter to absorb it would be invisible.
93+
94+
In update mode, a leg that disagrees with an existing shared golden **fails**
95+
instead of overwriting it — otherwise the last runtime to run would silently
96+
define the expectation for all of them.
97+
98+
## Pinned runtime infrastructure
99+
100+
The harness pins AWS Runtime Interface Emulator (RIE) `v1.36` and verifies the
101+
cached binary on every run before mounting it into a container:
102+
103+
| Asset | SHA-256 |
104+
|---|---|
105+
| `aws-lambda-rie-x86_64` | `ba57f2683260127135ad5ba9bafea141f90492143cbaeb9312cde6dae8d1c08e` |
106+
| `aws-lambda-rie-arm64` | `7826415f278663274e279085ff96d7c9da210a30213fa72279e56e59f028ce76` |
107+
108+
If `./bin/aws-lambda-rie` is absent or has a different checksum, `run.sh`
109+
downloads and verifies the platform-specific asset before replacing the cache.
110+
111+
Node 26 is still preview-only in ECR Public: the bare
112+
`public.ecr.aws/lambda/nodejs:26` tag does not exist. The logical runtime stays
113+
`26` for image names, function names, and snapshot paths, while the Docker
114+
base-image build argument maps to the dated multi-arch tag
115+
`26-preview.2026.08.21.22`.
116+
117+
Node 26 is a strict leg like every other: its normalized output matches the
118+
shared goldens exactly, so it needs no snapshots of its own.
119+
120+
When AWS publishes the bare Node 26 GA image, swap the pinned tag and re-run.
121+
If GA output diverges, add a `*_node26` override captured from the pinned
122+
pre-migration ref rather than absorbing the difference into `normalize.sh` —
123+
the oracle is tied to the implementation under test, and a base-image change
124+
must be reviewed, not hidden by normalization.
125+
126+
## Proactive-initialization simulation (manual diagnostic, not a gate)
127+
128+
**Nothing uses this.** It is not set in CI, not set in the default run, and
129+
nothing gates on it. A run with it is **expected to differ** from the goldens
130+
rather than match them, so it is a tool for reproducing a sandbox state by
131+
hand and reading the diff — not a pass/fail check. It also cannot affect the
132+
baseline: in update mode the shared-golden guard rejects a proactive run
133+
instead of overwriting. Update this section if it ever becomes part of the
134+
gate.
135+
136+
```bash
137+
SIMULATE_PROACTIVE_INIT=true RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh
138+
```
139+
140+
The library stamps `initTime = Date.now()` at wrapper-module load
141+
(`src/index.ts`) and marks the sandbox as proactively initialized when the
142+
first invocation starts more than 10 s after that
143+
(`src/utils/cold-start.ts`). On real Lambda, AWS sometimes runs the init
144+
phase well before the first invoke (proactive initialization), flipping that
145+
flag.
146+
147+
Reproducing this under RIE takes two ingredients, because **the classic RIE
148+
runs the init phase lazily on the first invocation** — a post-start `sleep`
149+
alone creates no init→invoke gap (verified: with only a sleep, the logs are
150+
byte-identical to an immediate run):
151+
152+
1. `run.sh` sets `AWS_LAMBDA_MAX_CONCURRENCY=1`, which switches the same RIE
153+
binary into its managed-instances path; that path performs the init phase
154+
(bootstrap + wrapper module load) **eagerly at container start**. Side
155+
effects are contained: the function env gains
156+
`AWS_LAMBDA_INITIALIZATION_TYPE=lambda-managed-instances`, which the
157+
library only uses to gate cold-start tracing spans (already disabled here
158+
via `DD_COLD_START_TRACING=false`), and `AWS_LAMBDA_LOG_FORMAT=text` is
159+
pinned to keep runtime logs in the classic text format.
160+
2. `run.sh` then sleeps 15 s before the first invocation, producing an
161+
init→invoke gap > 10 s.
162+
163+
The first invocation then emits, exactly as in the CI failure on
164+
`container-esm_node18`:
165+
166+
- `"proactive_initialization": 1` in the `aws.lambda` span's `metrics`,
167+
- `"proactive_initialization:true"` in the `aws.lambda.enhanced.invocations`
168+
metric tags,
169+
- `cold_start:false` (instead of `true`) on the first invocation.
170+
171+
A `SIMULATE_PROACTIVE_INIT=true` run is therefore **expected to fail** the
172+
diff against the immediate-run snapshots, with precisely those lines added
173+
(plus RIE-mode platform-log noise: `INIT/REPORT` line shapes and the emulated
174+
region/account in ARNs differ between the two RIE paths). The platform-side
175+
half of the CI diff, `END Duration: XXXX ms (init: XXXX ms)`, is emitted by
176+
the real Lambda platform and does not appear under RIE in either mode.
177+
178+
## Files
179+
180+
- `run.sh` — the runner (build images, run under RIE, invoke, diff snapshots)
181+
- `normalize.sh` — the local log-normalization pipeline, based on the AWS
182+
suite's filters with documented RIE-specific handling. Reads stdin, writes
183+
stdout; honors `RUN_ID` for optional per-run ID stripping.
184+
- `bin/` — downloaded RIE binary (gitignored)
185+
- `snapshots/logs/` — normalized log snapshots, shared per variant across
186+
runtimes, with optional `${variant}_node${major}.log` overrides
187+
- `snapshots/return_values/` — handler return-value snapshot, shared by every
188+
case, with optional `${variant}_node${major}_${event}.json` overrides
189+
190+
## Comparison with the AWS-based suite
191+
192+
| | AWS suite (`scripts/run_integration_tests.sh`) | this harness |
193+
|---|---|---|
194+
| handlers | 9 variants (layers + container images) | container images only (cjs, esm) |
195+
| infra | real Lambda via serverless, CloudWatch logs | docker + RIE, `docker logs` |
196+
| snapshots | `integration_tests/snapshots/` | `integration_tests_local/snapshots/` |
197+
| credentials | AWS account + DD_API_KEY | none |
198+
| cost/wait | deploy + invoke + 20 s log wait | image build + invoke |
199+
200+
Local snapshots legitimately differ from the AWS ones (fake account/region
201+
context, no real API Gateway IDs, RIE-formatted `START`/`END`/`REPORT`
202+
lines, no platform `init:` duration suffix). Do not diff one suite's output
203+
against the other's snapshots.
204+
205+
## Known emulation gaps (RIE vs real Lambda)
206+
207+
- The classic RIE runs the init phase **lazily on the first invocation**, so
208+
init→invoke timing behavior (proactive initialization) cannot be observed
209+
in the default mode; use `SIMULATE_PROACTIVE_INIT=true`, which switches
210+
RIE to its eager-init managed-instances path (see above).
211+
- No platform `INIT_START` / `END ... (init: N ms)` lines — those come from
212+
the Lambda platform, not the runtime.
213+
- No real AWS service context: API Gateway/DynamoDB/S3/SNS/SQS resource ARNs,
214+
account IDs, and inferred-span metadata are derived only from the event
215+
payloads, so they differ from the AWS snapshots.
216+
- `AWS_REGION` is faked to `eu-west-1` to keep the enhanced-metric region
217+
tag stable.
218+
- Enhanced metrics that depend on platform-provided values (e.g. real
219+
memory size / billed duration) may be absent or differ.
220+
- Onboarding-mode coverage is partial, and it is two modes short, not three.
221+
Both container fixtures end in
222+
`CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]`, so the
223+
**npm-installed** handler path is already exercised on every runtime. What
224+
this harness does not cover is the **layer** path
225+
(`/opt/nodejs/node_modules/datadog-lambda-js/handler.handler`, which needs the
226+
built layer zip and a zip-based deployment model) and **manual**
227+
`datadog(handler)` wrapping in customer code. Adding those two means new
228+
fixtures and new goldens, so they belong to a separately scoped follow-up
229+
rather than to the harness-hardening PR.
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/bin/bash
2+
3+
# Local log-normalization pipeline for integration test snapshots.
4+
#
5+
# This is based on the filter chain used by scripts/run_integration_tests.sh
6+
# to replace invocation-specific data (timestamps, IDs, durations, ...) with
7+
# XXXX before diffing logs against snapshots. Local-only RIE filters and
8+
# deterministic cold-start assertions are documented below.
9+
#
10+
# Usage:
11+
# some-log-producer | ./normalize.sh
12+
# RUN_ID=abcdef12 ./normalize.sh < raw.log > normalized.log
13+
#
14+
# Reads from stdin, writes to stdout.
15+
# Optional env:
16+
# RUN_ID - random per-run ID embedded in deployed function names; stripped
17+
# to XXXX when set (AWS suite only; the local harness uses static
18+
# function names and leaves this unset).
19+
20+
set -e
21+
22+
script_path=${BASH_SOURCE[0]}
23+
local_dir=$(dirname "$script_path")
24+
repo_dir=$(dirname "$local_dir")
25+
26+
# An empty Perl program is a pass-through when RUN_ID is unset.
27+
run_id_filter=''
28+
if [ -n "$RUN_ID" ]; then
29+
run_id_filter="s/${RUN_ID}/XXXX/g"
30+
fi
31+
32+
node "$repo_dir/integration_tests/parse-json.js" |
33+
# Filter serverless cli errors
34+
sed '/Serverless: Recoverable error occurred/d' |
35+
# Normalize Lambda runtime report logs
36+
perl -p -e 's/(RequestId|TraceId|init|SegmentId|Duration|Memory Used|"e"):( )?[a-z0-9\.\-]+/\1:\2XXXX/g' |
37+
# Node.js 26 preview and container-image runtimes emit extra platform noise.
38+
sed '/preview runtime version and should not be used for production workloads/d' |
39+
sed '/^INIT_REPORT /d' |
40+
sed '/DEP0205.*module\.register()/d' |
41+
sed '/node --trace-deprecation.*where the warning was created/d' |
42+
# Normalize DD APM headers and AWS account ID
43+
perl -p -e "s/(x-datadog-parent-id:|x-datadog-trace-id:|account_id:)[0-9]+/\1XXXX/g" |
44+
# Strip API key from logged requests
45+
perl -p -e "s/(api_key=|'api_key': ')[a-z0-9\.\-]+/\1XXXX/g" |
46+
# Normalize log timestamps
47+
perl -p -e "s/[0-9]{4}\-[0-9]{2}\-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]+( \(\-?\+?[0-9:]+\))?/XXXX-XX-XX XX:XX:XX.XXX/" |
48+
# Normalize DD trace ID injection
49+
perl -p -e "s/(dd\.trace_id=)[0-9]+ (dd\.span_id=)[0-9]+/\1XXXX \2XXXX/" |
50+
# Normalize execution ID in logs prefix
51+
perl -p -e $'s/[0-9a-z]+\-[0-9a-z]+\-[0-9a-z]+\-[0-9a-z]+\-[0-9a-z]+\t/XXXX-XXXX-XXXX-XXXX-XXXX\t/' |
52+
# Normalize minor package version tag so that these snapshots aren't broken on version bumps
53+
perl -p -e "s/(dd_lambda_layer:datadog-nodev[0-9]+\.)[0-9]+\.[0-9]+/\1XX\.X/g" |
54+
perl -p -e 's/"(span_id|apiid|runtime-id|record_ids|parent_id|trace_id|start|duration|tcp\.local\.address|tcp\.local\.port|dns\.address|request_id|function_arn|x-datadog-trace-id|x-datadog-parent-id|datadog_lambda|dd_trace|process_id)":\ ("?)[a-zA-Z0-9\.:\-]+("?)/"\1":\2XXXX\3/g' |
55+
# Strip out run ID (from function name, resource, etc.)
56+
perl -p -e "$run_id_filter" |
57+
# Normalize line numbers in stack traces
58+
perl -p -e 's/(.js:)[0-9]*:[0-9]*/\1XXX:XXX/g' |
59+
# Preserve stable span meta and metrics; volatile values are normalized above.
60+
# Normalize enhanced metric datadog_lambda tag
61+
perl -p -e "s/(datadog_lambda:v)[0-9\.]+/\1X.X.X/g" |
62+
# Normalize lookup resource
63+
perl -p -e "s/(\"resource\":\"169.)[0-9\.]+/\1X.X.X/g" |
64+
# Normalize Axios version
65+
perl -p -e "s/User-Agent:axios\/\d+\.\d+\.\d+/User-Agent:axios\/X\.X\.X/g" |
66+
# Remove init start line
67+
perl -p -e "s/INIT_START.*//g" |
68+
# Proactive initialization is platform scheduling, not code behavior.
69+
# Keep cold_start to cover the deterministic local cold-to-warm transition.
70+
sed '/proactive_initialization/d' |
71+
perl -p -e 's/ \(init: [^)]*\)//g' |
72+
# Normalize RIE platform log lines (local harness only; no-op on AWS logs):
73+
# "28 Jul 2026 19:42:34,536 [INFO] (rapid) ..." timestamps, request ids,
74+
# and init/invoke durations vary run to run
75+
perl -p -e 's/^[0-9]{2} \w{3} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} (\[INFO\] \(rapid\))/XXXX \1/' |
76+
perl -p -e 's/(requestId: )[0-9a-f-]+/\1XXXX/g' |
77+
perl -p -e 's/(duration(Ms)?: )[0-9.]+/\1XXXX/g' |
78+
sed -E "s/(tracestate\:)([A-Za-z0-9\-\=\:\;].+)/\1XXX/g" |
79+
sed -E "s/(\"_dd.p.tid\"\: \")[a-z0-9\.\-]+/\1XXXX/g" |
80+
sed -E "s/(_dd.p.tid=)[a-z0-9\.\-]+/\1XXXX/g" |
81+
# Remove RIE's trailing REPORT tab without changing application output.
82+
sed -E '/^REPORT RequestId:/s/[[:blank:]]+$//'

0 commit comments

Comments
 (0)