Crashtracker: dual ship to error tracking #10000

amarziali · 2025-11-19T13:59:14Z

What Does This Do

This PR introduces dual shipping of crash reports to both the telemetry and error-tracking intakes. While only the stack-frame section of the payload is shared between the two destinations, the remaining fields have been adapted to each intake’s expected format.

We improved signal-info (sig_info) parsing to extract the program counter address more reliably, and this information is now also included in the telemetry payload. Additional handling has been added for cases where an hs_err file indicates an OutOfMemory crash, as these reports typically lack signal data.

Communication Improvements

The OkHttp client has been configured to support up to four concurrent operations — telemetry ping, telemetry upload, error-tracking ping, and error-tracking upload. Similar to profiling, the underlying thread pool has been customized so that threads run as daemon threads, allowing the JVM to exit cleanly without shutdown delays.

Feature enablement

Uploading the crash info to error tracking is disabled by default. The feature can be enabled by either:

Setting the system property dd.crashtracking.errors-intake.enabled=true
Setting the env variable DD_CRASHTRACKING_ERRORS_INTAKE_ENABLED=true

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-11-19T14:48:55Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/dualship
git_commit_date	1763731854	1764079325
git_commit_sha	`614cc72`	`6150309`
release_version	1.57.0-SNAPSHOT~614cc721f6	1.57.0-SNAPSHOT~6150309b46

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1764081253	1764081253
ci_job_id	1255898054	1255898054
ci_pipeline_id	83939170	83939170
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-dzdwdtyi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-dzdwdtyi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 6 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.099 s) : 0, 1099350
Total [baseline] (8.876 s) : 0, 8875898
Agent [candidate] (1.098 s) : 0, 1098137
Total [candidate] (8.865 s) : 0, 8865133
section iast
Agent [baseline] (1.24 s) : 0, 1240444
Total [baseline] (9.595 s) : 0, 9594989
Agent [candidate] (1.247 s) : 0, 1247353
Total [candidate] (9.571 s) : 0, 9571172

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.099 s	-
Agent	iast	1.24 s	141.094 ms (12.8%)
Total	tracing	8.876 s	-
Total	iast	9.595 s	719.091 ms (8.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.098 s	-
Agent	iast	1.247 s	149.216 ms (13.6%)
Total	tracing	8.865 s	-
Total	iast	9.571 s	706.039 ms (8.0%)

gantt
    title insecure-bank - break down per module: candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.49 ms) : 0, 1490
crashtracking [candidate] (1.491 ms) : 0, 1491
BytebuddyAgent [baseline] (707.264 ms) : 0, 707264
BytebuddyAgent [candidate] (706.588 ms) : 0, 706588
GlobalTracer [baseline] (248.424 ms) : 0, 248424
GlobalTracer [candidate] (248.323 ms) : 0, 248323
AppSec [baseline] (31.993 ms) : 0, 31993
AppSec [candidate] (31.807 ms) : 0, 31807
Debugger [baseline] (62.796 ms) : 0, 62796
Debugger [candidate] (62.702 ms) : 0, 62702
Remote Config [baseline] (615.678 µs) : 0, 616
Remote Config [candidate] (619.395 µs) : 0, 619
Telemetry [baseline] (8.269 ms) : 0, 8269
Telemetry [candidate] (8.143 ms) : 0, 8143
Flare Poller [baseline] (3.668 ms) : 0, 3668
Flare Poller [candidate] (3.624 ms) : 0, 3624
section iast
crashtracking [baseline] (1.484 ms) : 0, 1484
crashtracking [candidate] (1.503 ms) : 0, 1503
BytebuddyAgent [baseline] (832.734 ms) : 0, 832734
BytebuddyAgent [candidate] (838.631 ms) : 0, 838631
GlobalTracer [baseline] (238.102 ms) : 0, 238102
GlobalTracer [candidate] (238.863 ms) : 0, 238863
IAST [baseline] (27.522 ms) : 0, 27522
IAST [candidate] (28.685 ms) : 0, 28685
AppSec [baseline] (34.087 ms) : 0, 34087
AppSec [candidate] (33.003 ms) : 0, 33003
Debugger [baseline] (59.898 ms) : 0, 59898
Debugger [candidate] (60.009 ms) : 0, 60009
Remote Config [baseline] (551.99 µs) : 0, 552
Remote Config [candidate] (545.657 µs) : 0, 546
Telemetry [baseline] (7.616 ms) : 0, 7616
Telemetry [candidate] (7.602 ms) : 0, 7602
Flare Poller [baseline] (3.497 ms) : 0, 3497
Flare Poller [candidate] (3.464 ms) : 0, 3464

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.103 s) : 0, 1103064
Total [baseline] (10.828 s) : 0, 10828123
Agent [candidate] (1.101 s) : 0, 1101085
Total [candidate] (10.795 s) : 0, 10794530
section appsec
Agent [baseline] (1.283 s) : 0, 1283028
Total [baseline] (11.159 s) : 0, 11159092
Agent [candidate] (1.285 s) : 0, 1284765
Total [candidate] (11.169 s) : 0, 11168652
section iast
Agent [baseline] (1.24 s) : 0, 1239824
Total [baseline] (11.304 s) : 0, 11304239
Agent [candidate] (1.239 s) : 0, 1239065
Total [candidate] (11.285 s) : 0, 11284668
section profiling
Agent [baseline] (1.261 s) : 0, 1260658
Total [baseline] (11.202 s) : 0, 11201844
Agent [candidate] (1.233 s) : 0, 1232761
Total [candidate] (11.104 s) : 0, 11103543

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.103 s	-
Agent	appsec	1.283 s	179.964 ms (16.3%)
Agent	iast	1.24 s	136.76 ms (12.4%)
Agent	profiling	1.261 s	157.594 ms (14.3%)
Total	tracing	10.828 s	-
Total	appsec	11.159 s	330.969 ms (3.1%)
Total	iast	11.304 s	476.117 ms (4.4%)
Total	profiling	11.202 s	373.721 ms (3.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.101 s	-
Agent	appsec	1.285 s	183.68 ms (16.7%)
Agent	iast	1.239 s	137.98 ms (12.5%)
Agent	profiling	1.233 s	131.676 ms (12.0%)
Total	tracing	10.795 s	-
Total	appsec	11.169 s	374.122 ms (3.5%)
Total	iast	11.285 s	490.138 ms (4.5%)
Total	profiling	11.104 s	309.012 ms (2.9%)

gantt
    title petclinic - break down per module: candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.48 ms) : 0, 1480
crashtracking [candidate] (1.483 ms) : 0, 1483
BytebuddyAgent [baseline] (708.54 ms) : 0, 708540
BytebuddyAgent [candidate] (707.196 ms) : 0, 707196
GlobalTracer [baseline] (249.0 ms) : 0, 249000
GlobalTracer [candidate] (249.264 ms) : 0, 249264
AppSec [baseline] (32.117 ms) : 0, 32117
AppSec [candidate] (31.913 ms) : 0, 31913
Debugger [baseline] (64.451 ms) : 0, 64451
Debugger [candidate] (63.88 ms) : 0, 63880
Remote Config [baseline] (629.694 µs) : 0, 630
Remote Config [candidate] (619.63 µs) : 0, 620
Telemetry [baseline] (8.321 ms) : 0, 8321
Telemetry [candidate] (8.274 ms) : 0, 8274
Flare Poller [baseline] (3.775 ms) : 0, 3775
Flare Poller [candidate] (3.707 ms) : 0, 3707
section appsec
crashtracking [baseline] (1.485 ms) : 0, 1485
crashtracking [candidate] (1.484 ms) : 0, 1484
BytebuddyAgent [baseline] (733.078 ms) : 0, 733078
BytebuddyAgent [candidate] (734.05 ms) : 0, 734050
GlobalTracer [baseline] (240.928 ms) : 0, 240928
GlobalTracer [candidate] (241.393 ms) : 0, 241393
AppSec [baseline] (173.534 ms) : 0, 173534
AppSec [candidate] (174.316 ms) : 0, 174316
Debugger [baseline] (61.609 ms) : 0, 61609
Debugger [candidate] (60.949 ms) : 0, 60949
Remote Config [baseline] (660.74 µs) : 0, 661
Remote Config [candidate] (667.434 µs) : 0, 667
Telemetry [baseline] (8.165 ms) : 0, 8165
Telemetry [candidate] (8.144 ms) : 0, 8144
Flare Poller [baseline] (4.028 ms) : 0, 4028
Flare Poller [candidate] (3.872 ms) : 0, 3872
IAST [baseline] (24.591 ms) : 0, 24591
IAST [candidate] (24.938 ms) : 0, 24938
section iast
crashtracking [baseline] (1.51 ms) : 0, 1510
crashtracking [candidate] (1.477 ms) : 0, 1477
BytebuddyAgent [baseline] (832.104 ms) : 0, 832104
BytebuddyAgent [candidate] (831.762 ms) : 0, 831762
GlobalTracer [baseline] (237.611 ms) : 0, 237611
GlobalTracer [candidate] (237.62 ms) : 0, 237620
AppSec [baseline] (31.239 ms) : 0, 31239
AppSec [candidate] (32.676 ms) : 0, 32676
Debugger [baseline] (60.572 ms) : 0, 60572
Debugger [candidate] (60.619 ms) : 0, 60619
Remote Config [baseline] (542.903 µs) : 0, 543
Remote Config [candidate] (534.48 µs) : 0, 534
Telemetry [baseline] (7.606 ms) : 0, 7606
Telemetry [candidate] (7.549 ms) : 0, 7549
Flare Poller [baseline] (3.441 ms) : 0, 3441
Flare Poller [candidate] (3.473 ms) : 0, 3473
IAST [baseline] (30.357 ms) : 0, 30357
IAST [candidate] (28.458 ms) : 0, 28458
section profiling
ProfilingAgent [baseline] (98.22 ms) : 0, 98220
ProfilingAgent [candidate] (96.761 ms) : 0, 96761
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.443 ms) : 0, 1443
BytebuddyAgent [baseline] (754.584 ms) : 0, 754584
BytebuddyAgent [candidate] (735.538 ms) : 0, 735538
GlobalTracer [baseline] (225.603 ms) : 0, 225603
GlobalTracer [candidate] (222.683 ms) : 0, 222683
AppSec [baseline] (33.096 ms) : 0, 33096
AppSec [candidate] (32.038 ms) : 0, 32038
Debugger [baseline] (64.425 ms) : 0, 64425
Debugger [candidate] (62.796 ms) : 0, 62796
Remote Config [baseline] (677.165 µs) : 0, 677
Remote Config [candidate] (687.114 µs) : 0, 687
Telemetry [baseline] (8.167 ms) : 0, 8167
Telemetry [candidate] (7.934 ms) : 0, 7934
Flare Poller [baseline] (3.824 ms) : 0, 3824
Flare Poller [candidate] (3.753 ms) : 0, 3753
Profiling [baseline] (98.815 ms) : 0, 98815
Profiling [candidate] (97.327 ms) : 0, 97327

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/dualship
git_commit_date	1763731854	1764079325
git_commit_sha	`614cc72`	`6150309`
release_version	1.57.0-SNAPSHOT~614cc721f6	1.57.0-SNAPSHOT~6150309b46

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1764081647	1764081647
ci_job_id	1255898056	1255898056
ci_pipeline_id	83939170	83939170
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-s3hspann 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-s3hspann 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 19 metrics, 16 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:petclinic:profiling:high_load	better [-1.788ms; -0.787ms] or [-9.156%; -4.031%]	unsure [-1.916ms; -0.194ms] or [-6.232%; -0.632%]	unstable [-15.330op/s; +29.356op/s] or [-6.268%; +12.002%]	18.237ms	29.682ms	251.594op/s	19.524ms	30.737ms	244.581op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.231 ms) : 1219, 1243
.   : milestone, 1231,
iast (3.261 ms) : 3216, 3305
.   : milestone, 3261,
iast_FULL (5.839 ms) : 5781, 5898
.   : milestone, 5839,
iast_GLOBAL (3.547 ms) : 3495, 3600
.   : milestone, 3547,
profiling (2.143 ms) : 2124, 2162
.   : milestone, 2143,
tracing (1.894 ms) : 1876, 1911
.   : milestone, 1894,
section candidate
no_agent (1.207 ms) : 1195, 1219
.   : milestone, 1207,
iast (3.288 ms) : 3244, 3331
.   : milestone, 3288,
iast_FULL (5.762 ms) : 5706, 5819
.   : milestone, 5762,
iast_GLOBAL (3.634 ms) : 3580, 3688
.   : milestone, 3634,
profiling (2.218 ms) : 2197, 2239
.   : milestone, 2218,
tracing (1.844 ms) : 1829, 1860
.   : milestone, 1844,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.231 ms [1.219 ms, 1.243 ms]	-
iast	3.261 ms [3.216 ms, 3.305 ms]	2.03 ms (165.0%)
iast_FULL	5.839 ms [5.781 ms, 5.898 ms]	4.609 ms (374.5%)
iast_GLOBAL	3.547 ms [3.495 ms, 3.6 ms]	2.317 ms (188.3%)
profiling	2.143 ms [2.124 ms, 2.162 ms]	912.447 µs (74.2%)
tracing	1.894 ms [1.876 ms, 1.911 ms]	663.067 µs (53.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.207 ms [1.195 ms, 1.219 ms]	-
iast	3.288 ms [3.244 ms, 3.331 ms]	2.081 ms (172.4%)
iast_FULL	5.762 ms [5.706 ms, 5.819 ms]	4.555 ms (377.4%)
iast_GLOBAL	3.634 ms [3.58 ms, 3.688 ms]	2.427 ms (201.0%)
profiling	2.218 ms [2.197 ms, 2.239 ms]	1.011 ms (83.7%)
tracing	1.844 ms [1.829 ms, 1.86 ms]	637.147 µs (52.8%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.35 ms) : 18159, 18541
.   : milestone, 18350,
appsec (18.719 ms) : 18528, 18909
.   : milestone, 18719,
code_origins (17.674 ms) : 17501, 17846
.   : milestone, 17674,
iast (18.112 ms) : 17929, 18294
.   : milestone, 18112,
profiling (19.699 ms) : 19501, 19897
.   : milestone, 19699,
tracing (17.63 ms) : 17455, 17805
.   : milestone, 17630,
section candidate
no_agent (17.305 ms) : 17132, 17477
.   : milestone, 17305,
appsec (18.461 ms) : 18274, 18647
.   : milestone, 18461,
code_origins (17.841 ms) : 17663, 18020
.   : milestone, 17841,
iast (17.778 ms) : 17600, 17957
.   : milestone, 17778,
profiling (18.551 ms) : 18362, 18739
.   : milestone, 18551,
tracing (17.6 ms) : 17428, 17772
.   : milestone, 17600,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.35 ms [18.159 ms, 18.541 ms]	-
appsec	18.719 ms [18.528 ms, 18.909 ms]	368.732 µs (2.0%)
code_origins	17.674 ms [17.501 ms, 17.846 ms]	-676.424 µs (-3.7%)
iast	18.112 ms [17.929 ms, 18.294 ms]	-238.436 µs (-1.3%)
profiling	19.699 ms [19.501 ms, 19.897 ms]	1.349 ms (7.4%)
tracing	17.63 ms [17.455 ms, 17.805 ms]	-720.007 µs (-3.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.305 ms [17.132 ms, 17.477 ms]	-
appsec	18.461 ms [18.274 ms, 18.647 ms]	1.156 ms (6.7%)
code_origins	17.841 ms [17.663 ms, 18.02 ms]	536.487 µs (3.1%)
iast	17.778 ms [17.6 ms, 17.957 ms]	473.73 µs (2.7%)
profiling	18.551 ms [18.362 ms, 18.739 ms]	1.246 ms (7.2%)
tracing	17.6 ms [17.428 ms, 17.772 ms]	295.234 µs (1.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/dualship
git_commit_date	1763731854	1764079325
git_commit_sha	`614cc72`	`6150309`
release_version	1.57.0-SNAPSHOT~614cc721f6	1.57.0-SNAPSHOT~6150309b46

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1764081433	1764081433
ci_job_id	1255898057	1255898057
ci_pipeline_id	83939170	83939170
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-hi0e5slo 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-hi0e5slo 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.487 ms) : 1476, 1499
.   : milestone, 1487,
appsec (2.464 ms) : 2412, 2515
.   : milestone, 2464,
iast (2.232 ms) : 2166, 2297
.   : milestone, 2232,
iast_GLOBAL (2.28 ms) : 2214, 2345
.   : milestone, 2280,
profiling (2.502 ms) : 2340, 2664
.   : milestone, 2502,
tracing (2.063 ms) : 2012, 2114
.   : milestone, 2063,
section candidate
no_agent (1.48 ms) : 1469, 1492
.   : milestone, 1480,
appsec (2.48 ms) : 2428, 2532
.   : milestone, 2480,
iast (2.224 ms) : 2159, 2289
.   : milestone, 2224,
iast_GLOBAL (2.271 ms) : 2206, 2336
.   : milestone, 2271,
profiling (2.116 ms) : 2061, 2170
.   : milestone, 2116,
tracing (2.056 ms) : 2005, 2107
.   : milestone, 2056,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.487 ms [1.476 ms, 1.499 ms]	-
appsec	2.464 ms [2.412 ms, 2.515 ms]	976.388 µs (65.6%)
iast	2.232 ms [2.166 ms, 2.297 ms]	744.428 µs (50.0%)
iast_GLOBAL	2.28 ms [2.214 ms, 2.345 ms]	792.185 µs (53.3%)
profiling	2.502 ms [2.34 ms, 2.664 ms]	1.015 ms (68.2%)
tracing	2.063 ms [2.012 ms, 2.114 ms]	575.715 µs (38.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.48 ms [1.469 ms, 1.492 ms]	-
appsec	2.48 ms [2.428 ms, 2.532 ms]	999.251 µs (67.5%)
iast	2.224 ms [2.159 ms, 2.289 ms]	743.522 µs (50.2%)
iast_GLOBAL	2.271 ms [2.206 ms, 2.336 ms]	790.266 µs (53.4%)
profiling	2.116 ms [2.061 ms, 2.17 ms]	635.344 µs (42.9%)
tracing	2.056 ms [2.005 ms, 2.107 ms]	575.995 µs (38.9%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~6150309b46, baseline=1.57.0-SNAPSHOT~614cc721f6
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.629 s) : 15629000, 15629000
.   : milestone, 15629000,
appsec (14.755 s) : 14755000, 14755000
.   : milestone, 14755000,
iast (18.505 s) : 18505000, 18505000
.   : milestone, 18505000,
iast_GLOBAL (18.354 s) : 18354000, 18354000
.   : milestone, 18354000,
profiling (14.598 s) : 14598000, 14598000
.   : milestone, 14598000,
tracing (14.92 s) : 14920000, 14920000
.   : milestone, 14920000,
section candidate
no_agent (15.364 s) : 15364000, 15364000
.   : milestone, 15364000,
appsec (14.657 s) : 14657000, 14657000
.   : milestone, 14657000,
iast (18.051 s) : 18051000, 18051000
.   : milestone, 18051000,
iast_GLOBAL (17.717 s) : 17717000, 17717000
.   : milestone, 17717000,
profiling (14.599 s) : 14599000, 14599000
.   : milestone, 14599000,
tracing (15.027 s) : 15027000, 15027000
.   : milestone, 15027000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.629 s [15.629 s, 15.629 s]	-
appsec	14.755 s [14.755 s, 14.755 s]	-874.0 ms (-5.6%)
iast	18.505 s [18.505 s, 18.505 s]	2.876 s (18.4%)
iast_GLOBAL	18.354 s [18.354 s, 18.354 s]	2.725 s (17.4%)
profiling	14.598 s [14.598 s, 14.598 s]	-1.031 s (-6.6%)
tracing	14.92 s [14.92 s, 14.92 s]	-709.0 ms (-4.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.364 s [15.364 s, 15.364 s]	-
appsec	14.657 s [14.657 s, 14.657 s]	-707.0 ms (-4.6%)
iast	18.051 s [18.051 s, 18.051 s]	2.687 s (17.5%)
iast_GLOBAL	17.717 s [17.717 s, 17.717 s]	2.353 s (15.3%)
profiling	14.599 s [14.599 s, 14.599 s]	-765.0 ms (-5.0%)
tracing	15.027 s [15.027 s, 15.027 s]	-337.0 ms (-2.2%)

pr-commenter · 2025-11-21T15:23:01Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1763737917	1763738252
end_time	2025-11-21T15:13:19	2025-11-21T15:18:55
git_branch	master	andrea.marziali/dualship
git_commit_sha	`614cc72`	`b921659`
start_time	2025-11-21T15:11:57	2025-11-21T15:17:33

See matching parameters

	Baseline	Candidate
ci_job_id	1250844207	1250844207
ci_pipeline_id	83600252	83600252
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1763736936	1763736936

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 7 metrics, 8 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-59.135µs; +34.084µs] or [-19.580%; +11.286%]	unstable [-73.753µs; +41.176µs] or [-21.331%; +11.909%]	unstable [-89.749µs; +51.609µs] or [-24.689%; +14.197%]	unstable [-703.419µs; +293.699µs] or [-59.843%; +24.986%]	unstable [-127.167op/s; +127.167op/s] or [-5.277%; +5.277%]
scenario:basic	same	same	same	unstable [-92.405µs; +176.369µs] or [-11.781%; +22.486%]	unstable [-127.260op/s; +255.465op/s] or [-5.090%; +10.219%]
scenario:loop	same	same	same	unstable [-322.103µs; +564.535µs] or [-3.695%; +6.477%]	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (345.758 µs) : 277, 414
.   : milestone, 346,
basic (296.369 µs) : 286, 307
.   : milestone, 296,
loop (8.442 ms) : 8380, 8504
.   : milestone, 8442,
section candidate
noprobe (329.469 µs) : 294, 365
.   : milestone, 329,
basic (294.3 µs) : 287, 302
.   : milestone, 294,
loop (8.44 ms) : 8379, 8501
.   : milestone, 8440,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	345.758 µs [277.278 µs, 414.238 µs]
basic	296.369 µs [285.882 µs, 306.857 µs]
loop	8.442 ms [8.38 ms, 8.504 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	329.469 µs [293.572 µs, 365.366 µs]
basic	294.3 µs [286.684 µs, 301.917 µs]
loop	8.44 ms [8.379 ms, 8.501 ms]

github-actions · 2025-11-24T09:16:38Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

gleocadie

LGTM

GianlucaBortoli

Shipping crashes to the error tracking intake should be opt in like in libdatadog via the usual config mechanisms.

But the rest of the logic looks good!

amarziali · 2025-11-25T13:53:09Z

Shipping crashes to the error tracking intake should be opt in like in libdatadog via the usual config mechanisms.

But the rest of the logic looks good!

Hi @GianlucaBortoli the feature has been made opt-in now. Thanks for the suggestion

amarziali added 2 commits November 21, 2025 16:29

Crashtracker: dual ship to error tracking

28cb436

Fix error tracking payload

0882768

amarziali force-pushed the andrea.marziali/dualship branch from b921659 to 0882768 Compare November 21, 2025 15:34

Handle OOM and clean exit

2dca270

amarziali marked this pull request as ready for review November 24, 2025 09:16

amarziali requested review from a team as code owners November 24, 2025 09:16

amarziali requested a review from dougqh November 24, 2025 09:16

amarziali requested review from gleocadie and jbachorik November 24, 2025 09:16

amarziali added type: enhancement Enhancements and improvements comp: crash tracking Crash Tracking labels Nov 24, 2025

amarziali requested a review from GianlucaBortoli November 24, 2025 09:19

gleocadie approved these changes Nov 24, 2025

View reviewed changes

amarziali added 2 commits November 24, 2025 10:47

leftover

24715ac

adjust smoke test

399f8cc

GianlucaBortoli requested changes Nov 24, 2025

View reviewed changes

amarziali requested a review from GianlucaBortoli November 25, 2025 13:53

Make error intake upload optin

6150309

GianlucaBortoli approved these changes Nov 25, 2025

View reviewed changes

amarziali force-pushed the andrea.marziali/dualship branch from 03b34b3 to 6150309 Compare November 25, 2025 14:02

amarziali requested a review from a team as a code owner November 25, 2025 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Crashtracker: dual ship to error tracking #10000

Crashtracker: dual ship to error tracking #10000

amarziali commented Nov 19, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Nov 19, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Nov 21, 2025

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

gleocadie left a comment

Uh oh!

GianlucaBortoli left a comment •

edited

Loading

Uh oh!

amarziali commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Crashtracker: dual ship to error tracking #10000

Are you sure you want to change the base?

Crashtracker: dual ship to error tracking #10000

Conversation

amarziali commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Communication Improvements

Feature enablement

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

pr-commenter bot commented Nov 21, 2025

Debugger benchmarks

Parameters

Summary

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

gleocadie left a comment

Choose a reason for hiding this comment

Uh oh!

GianlucaBortoli left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amarziali commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amarziali commented Nov 19, 2025 •

edited

Loading

pr-commenter bot commented Nov 19, 2025 •

edited

Loading

GianlucaBortoli left a comment •

edited

Loading