|
| 1 | +# Local integration tests (docker + AWS RIE, no AWS account required) |
| 2 | + |
| 3 | +This directory contains a **local** integration test harness for |
| 4 | +datadog-lambda-js. It runs the container-image handler variants |
| 5 | +(`integration_tests/container/cjs` and `integration_tests/container/esm`) |
| 6 | +inside Docker against the |
| 7 | +[AWS Lambda Runtime Interface Emulator (RIE)](https://github.com/aws/aws-lambda-runtime-interface-emulator), |
| 8 | +invokes them with the same input events as the AWS-based suite, captures |
| 9 | +logs from `docker logs`, normalizes them with `./normalize.sh` — the AWS |
| 10 | +suite's filter chain plus documented RIE-specific handling, not an identical |
| 11 | +copy — and diffs them against **local** snapshots in `./snapshots/`. |
| 12 | + |
| 13 | +Nothing here touches AWS, and nothing here touches |
| 14 | +`integration_tests/snapshots/` (the AWS suite's snapshots). |
| 15 | + |
| 16 | +## Prerequisites |
| 17 | + |
| 18 | +- Docker (tested with colima on macOS arm64) |
| 19 | +- `node`, `yarn`, `perl`, `sed`, `curl` (all already used by the AWS suite) |
| 20 | +- Network access for the first run (pulls `public.ecr.aws/lambda/nodejs:*` |
| 21 | + base images, ~1 GB each, and downloads the pinned RIE binary into `./bin/`) |
| 22 | + |
| 23 | +## Running |
| 24 | + |
| 25 | +```bash |
| 26 | +# Everything: nodejs 18/20/22/24/26 x {cjs, esm} |
| 27 | +./integration_tests_local/run.sh |
| 28 | + |
| 29 | +# One runtime / one variant |
| 30 | +RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh |
| 31 | + |
| 32 | +# (Re)generate local snapshots |
| 33 | +UPDATE_SNAPSHOTS=true ./integration_tests_local/run.sh |
| 34 | + |
| 35 | +# Skip repacking the library (reuse container/*/datadog-lambda-js-local.tgz) |
| 36 | +SKIP_PACK=true RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh |
| 37 | + |
| 38 | +# Force amd64 images instead of arm64 |
| 39 | +PLATFORM=linux/amd64 ./integration_tests_local/run.sh |
| 40 | +``` |
| 41 | + |
| 42 | +Unless `SKIP_PACK=true` is set, each run repacks the library under test |
| 43 | +(`yarn install --frozen-lockfile && yarn build && npm pack`) into |
| 44 | +`integration_tests/container/{cjs,esm}/datadog-lambda-js-local.tgz`, exactly |
| 45 | +like `scripts/run_integration_tests.sh` does, so the containers always test |
| 46 | +the working tree. |
| 47 | + |
| 48 | +The harness takes no flags or positional arguments — configuration is |
| 49 | +environment-only, and an unexpected argument is rejected rather than ignored. |
| 50 | + |
| 51 | +Without `UPDATE_SNAPSHOTS=true`, every expected return-value and log snapshot |
| 52 | +must already exist. A missing snapshot fails the run and is never created |
| 53 | +implicitly. Update mode is the only path that creates or overwrites snapshots. |
| 54 | + |
| 55 | +## Snapshot layout: shared goldens with per-case overrides |
| 56 | + |
| 57 | +There are three committed snapshots, not one per case: |
| 58 | + |
| 59 | +``` |
| 60 | +snapshots/return_values/default.json # every event, every runtime, every variant |
| 61 | +snapshots/logs/cjs.log # nodejs 18/20/22/24/26, cjs |
| 62 | +snapshots/logs/esm.log # nodejs 18/20/22/24/26, esm |
| 63 | +``` |
| 64 | + |
| 65 | +All 9 input events return the same fixture response, and the normalized logs |
| 66 | +are identical across all five runtimes, so a per-case file per leg would be 80 |
| 67 | +copies of 3 distinct expectations — and 80 files to review when one of them |
| 68 | +legitimately changes. |
| 69 | + |
| 70 | +Two things make the sharing safe rather than lossy: |
| 71 | + |
| 72 | +- **The runtime is asserted before it is collapsed.** `runtime:nodejsNN.x` is |
| 73 | + the only genuinely runtime-specific line in the log. `run.sh` checks that it |
| 74 | + appears on every invocation with the major actually under test, and only then |
| 75 | + rewrites it to `nodejsXX.x`. Sharing the golden therefore does not stop the |
| 76 | + suite from checking that the library reports the runtime it is running on. |
| 77 | +- **`AWS_LAMBDA_FUNCTION_NAME` carries no runtime major.** It propagates into |
| 78 | + `service`, `resource`, `resource_names`, `functionname`, `function_arn`, |
| 79 | + `_dd.base_service` and `_dd.tags.process`, so embedding the runtime there |
| 80 | + made every runtime's golden differ in ~100 lines of pure fixture naming, |
| 81 | + burying the one line that was actually runtime-specific. |
| 82 | + |
| 83 | +Divergence is expressed by **adding a file**, never by loosening a comparison: |
| 84 | + |
| 85 | +``` |
| 86 | +snapshots/logs/${variant}_node${major}.log # overrides the shared log |
| 87 | +snapshots/return_values/${variant}_node${major}_${event}.json # overrides the shared return value |
| 88 | +``` |
| 89 | + |
| 90 | +When an override is present it wins for that leg alone. This keeps a real |
| 91 | +behavioral difference visible in review, whereas widening a normalization |
| 92 | +filter to absorb it would be invisible. |
| 93 | + |
| 94 | +In update mode, a leg that disagrees with an existing shared golden **fails** |
| 95 | +instead of overwriting it — otherwise the last runtime to run would silently |
| 96 | +define the expectation for all of them. |
| 97 | + |
| 98 | +## Pinned runtime infrastructure |
| 99 | + |
| 100 | +The harness pins AWS Runtime Interface Emulator (RIE) `v1.36` and verifies the |
| 101 | +cached binary on every run before mounting it into a container: |
| 102 | + |
| 103 | +| Asset | SHA-256 | |
| 104 | +|---|---| |
| 105 | +| `aws-lambda-rie-x86_64` | `ba57f2683260127135ad5ba9bafea141f90492143cbaeb9312cde6dae8d1c08e` | |
| 106 | +| `aws-lambda-rie-arm64` | `7826415f278663274e279085ff96d7c9da210a30213fa72279e56e59f028ce76` | |
| 107 | + |
| 108 | +If `./bin/aws-lambda-rie` is absent or has a different checksum, `run.sh` |
| 109 | +downloads and verifies the platform-specific asset before replacing the cache. |
| 110 | + |
| 111 | +Node 26 is still preview-only in ECR Public: the bare |
| 112 | +`public.ecr.aws/lambda/nodejs:26` tag does not exist. The logical runtime stays |
| 113 | +`26` for image names, function names, and snapshot paths, while the Docker |
| 114 | +base-image build argument maps to the dated multi-arch tag |
| 115 | +`26-preview.2026.08.21.22`. |
| 116 | + |
| 117 | +Node 26 is a strict leg like every other: its normalized output matches the |
| 118 | +shared goldens exactly, so it needs no snapshots of its own. |
| 119 | + |
| 120 | +When AWS publishes the bare Node 26 GA image, swap the pinned tag and re-run. |
| 121 | +If GA output diverges, add a `*_node26` override captured from the pinned |
| 122 | +pre-migration ref rather than absorbing the difference into `normalize.sh` — |
| 123 | +the oracle is tied to the implementation under test, and a base-image change |
| 124 | +must be reviewed, not hidden by normalization. |
| 125 | + |
| 126 | +## Proactive-initialization simulation (manual diagnostic, not a gate) |
| 127 | + |
| 128 | +**Nothing uses this.** It is not set in CI, not set in the default run, and |
| 129 | +nothing gates on it. A run with it is **expected to differ** from the goldens |
| 130 | +rather than match them, so it is a tool for reproducing a sandbox state by |
| 131 | +hand and reading the diff — not a pass/fail check. It also cannot affect the |
| 132 | +baseline: in update mode the shared-golden guard rejects a proactive run |
| 133 | +instead of overwriting. Update this section if it ever becomes part of the |
| 134 | +gate. |
| 135 | + |
| 136 | +```bash |
| 137 | +SIMULATE_PROACTIVE_INIT=true RUNTIME_PARAM=18 VARIANT_PARAM=esm ./integration_tests_local/run.sh |
| 138 | +``` |
| 139 | + |
| 140 | +The library stamps `initTime = Date.now()` at wrapper-module load |
| 141 | +(`src/index.ts`) and marks the sandbox as proactively initialized when the |
| 142 | +first invocation starts more than 10 s after that |
| 143 | +(`src/utils/cold-start.ts`). On real Lambda, AWS sometimes runs the init |
| 144 | +phase well before the first invoke (proactive initialization), flipping that |
| 145 | +flag. |
| 146 | + |
| 147 | +Reproducing this under RIE takes two ingredients, because **the classic RIE |
| 148 | +runs the init phase lazily on the first invocation** — a post-start `sleep` |
| 149 | +alone creates no init→invoke gap (verified: with only a sleep, the logs are |
| 150 | +byte-identical to an immediate run): |
| 151 | + |
| 152 | +1. `run.sh` sets `AWS_LAMBDA_MAX_CONCURRENCY=1`, which switches the same RIE |
| 153 | + binary into its managed-instances path; that path performs the init phase |
| 154 | + (bootstrap + wrapper module load) **eagerly at container start**. Side |
| 155 | + effects are contained: the function env gains |
| 156 | + `AWS_LAMBDA_INITIALIZATION_TYPE=lambda-managed-instances`, which the |
| 157 | + library only uses to gate cold-start tracing spans (already disabled here |
| 158 | + via `DD_COLD_START_TRACING=false`), and `AWS_LAMBDA_LOG_FORMAT=text` is |
| 159 | + pinned to keep runtime logs in the classic text format. |
| 160 | +2. `run.sh` then sleeps 15 s before the first invocation, producing an |
| 161 | + init→invoke gap > 10 s. |
| 162 | + |
| 163 | +The first invocation then emits, exactly as in the CI failure on |
| 164 | +`container-esm_node18`: |
| 165 | + |
| 166 | +- `"proactive_initialization": 1` in the `aws.lambda` span's `metrics`, |
| 167 | +- `"proactive_initialization:true"` in the `aws.lambda.enhanced.invocations` |
| 168 | + metric tags, |
| 169 | +- `cold_start:false` (instead of `true`) on the first invocation. |
| 170 | + |
| 171 | +A `SIMULATE_PROACTIVE_INIT=true` run is therefore **expected to fail** the |
| 172 | +diff against the immediate-run snapshots, with precisely those lines added |
| 173 | +(plus RIE-mode platform-log noise: `INIT/REPORT` line shapes and the emulated |
| 174 | +region/account in ARNs differ between the two RIE paths). The platform-side |
| 175 | +half of the CI diff, `END Duration: XXXX ms (init: XXXX ms)`, is emitted by |
| 176 | +the real Lambda platform and does not appear under RIE in either mode. |
| 177 | + |
| 178 | +## Files |
| 179 | + |
| 180 | +- `run.sh` — the runner (build images, run under RIE, invoke, diff snapshots) |
| 181 | +- `normalize.sh` — the local log-normalization pipeline, based on the AWS |
| 182 | + suite's filters with documented RIE-specific handling. Reads stdin, writes |
| 183 | + stdout; honors `RUN_ID` for optional per-run ID stripping. |
| 184 | +- `bin/` — downloaded RIE binary (gitignored) |
| 185 | +- `snapshots/logs/` — normalized log snapshots, shared per variant across |
| 186 | + runtimes, with optional `${variant}_node${major}.log` overrides |
| 187 | +- `snapshots/return_values/` — handler return-value snapshot, shared by every |
| 188 | + case, with optional `${variant}_node${major}_${event}.json` overrides |
| 189 | + |
| 190 | +## Comparison with the AWS-based suite |
| 191 | + |
| 192 | +| | AWS suite (`scripts/run_integration_tests.sh`) | this harness | |
| 193 | +|---|---|---| |
| 194 | +| handlers | 9 variants (layers + container images) | container images only (cjs, esm) | |
| 195 | +| infra | real Lambda via serverless, CloudWatch logs | docker + RIE, `docker logs` | |
| 196 | +| snapshots | `integration_tests/snapshots/` | `integration_tests_local/snapshots/` | |
| 197 | +| credentials | AWS account + DD_API_KEY | none | |
| 198 | +| cost/wait | deploy + invoke + 20 s log wait | image build + invoke | |
| 199 | + |
| 200 | +Local snapshots legitimately differ from the AWS ones (fake account/region |
| 201 | +context, no real API Gateway IDs, RIE-formatted `START`/`END`/`REPORT` |
| 202 | +lines, no platform `init:` duration suffix). Do not diff one suite's output |
| 203 | +against the other's snapshots. |
| 204 | + |
| 205 | +## Known emulation gaps (RIE vs real Lambda) |
| 206 | + |
| 207 | +- The classic RIE runs the init phase **lazily on the first invocation**, so |
| 208 | + init→invoke timing behavior (proactive initialization) cannot be observed |
| 209 | + in the default mode; use `SIMULATE_PROACTIVE_INIT=true`, which switches |
| 210 | + RIE to its eager-init managed-instances path (see above). |
| 211 | +- No platform `INIT_START` / `END ... (init: N ms)` lines — those come from |
| 212 | + the Lambda platform, not the runtime. |
| 213 | +- No real AWS service context: API Gateway/DynamoDB/S3/SNS/SQS resource ARNs, |
| 214 | + account IDs, and inferred-span metadata are derived only from the event |
| 215 | + payloads, so they differ from the AWS snapshots. |
| 216 | +- `AWS_REGION` is faked to `eu-west-1` to keep the enhanced-metric region |
| 217 | + tag stable. |
| 218 | +- Enhanced metrics that depend on platform-provided values (e.g. real |
| 219 | + memory size / billed duration) may be absent or differ. |
| 220 | +- Onboarding-mode coverage is partial, and it is two modes short, not three. |
| 221 | + Both container fixtures end in |
| 222 | + `CMD ["node_modules/datadog-lambda-js/dist/handler.handler"]`, so the |
| 223 | + **npm-installed** handler path is already exercised on every runtime. What |
| 224 | + this harness does not cover is the **layer** path |
| 225 | + (`/opt/nodejs/node_modules/datadog-lambda-js/handler.handler`, which needs the |
| 226 | + built layer zip and a zip-based deployment model) and **manual** |
| 227 | + `datadog(handler)` wrapping in customer code. Adding those two means new |
| 228 | + fixtures and new goldens, so they belong to a separately scoped follow-up |
| 229 | + rather than to the harness-hardening PR. |
0 commit comments