Fix pekko instrumentation in a rare edge case #9676

vandonr · 2025-10-06T14:31:01Z

What Does This Do

pekko tests on error handling are flaky, and Andrea identified the issue in the logs as (probably) a span that has a continuation that is left open. We see this from this log: PendingTrace enqueued but won't be reported.
Thus, the span is not written immediately, and since the test that is failing is waiting for the trace with a timeout, we don't see the trace we expect within the set timeframe.

Since I have no experience with Pekko or that instrumentation, I called an AI for help, which highlighted this comment:

dd-trace-java/dd-java-agent/instrumentation/pekko/pekko-http-1.0/src/main/java/datadog/trace/instrumentation/pekkohttp/DatadogServerRequestResponseFlowWrapper.java

Lines 70 to 74 in f99014d

    
           // Since we haven't instrumented the pekko stream state machine, we can't rely 
        
           // on spans and scopes being propagated during the push and pull of the 
        
           // element. Instead we let the scope leak intentionally here and clean it 
        
           // up when the user response comes back, or in the actor message processing 
        
           // instrumentation that drives this state machine.

as a potential source for the lost continuation. It then suggested fixing the onUpstreamFailure handler, which is what I'm pushing here.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-10-06T14:34:20Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	vandonr/fix
git_commit_date	1759752550	1759758855
git_commit_sha	`20dac8a`	`f99014d`
release_version	1.54.0-SNAPSHOT~20dac8a306	1.54.0-SNAPSHOT~f99014d8ad

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1759760994	1759760994
ci_job_id	1164768979	1164768979
ci_pipeline_id	78473758	78473758
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-97wodehi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-97wodehi 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 8 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1025613
Total [baseline] (8.626 s) : 0, 8625552
Agent [candidate] (1.021 s) : 0, 1020754
Total [candidate] (8.656 s) : 0, 8655927
section iast
Agent [baseline] (1.161 s) : 0, 1160634
Total [baseline] (9.261 s) : 0, 9261225
Agent [candidate] (1.152 s) : 0, 1152412
Total [candidate] (9.215 s) : 0, 9215059

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.026 s	-
Agent	iast	1.161 s	135.021 ms (13.2%)
Total	tracing	8.626 s	-
Total	iast	9.261 s	635.673 ms (7.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.021 s	-
Agent	iast	1.152 s	131.657 ms (12.9%)
Total	tracing	8.656 s	-
Total	iast	9.215 s	559.132 ms (6.5%)

gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.455 ms) : 0, 1455
BytebuddyAgent [baseline] (699.543 ms) : 0, 699543
BytebuddyAgent [candidate] (694.767 ms) : 0, 694767
GlobalTracer [baseline] (243.086 ms) : 0, 243086
GlobalTracer [candidate] (243.064 ms) : 0, 243064
AppSec [baseline] (33.059 ms) : 0, 33059
AppSec [candidate] (32.928 ms) : 0, 32928
Debugger [baseline] (6.457 ms) : 0, 6457
Debugger [candidate] (6.421 ms) : 0, 6421
Remote Config [baseline] (695.333 µs) : 0, 695
Remote Config [candidate] (698.614 µs) : 0, 699
Telemetry [baseline] (9.416 ms) : 0, 9416
Telemetry [candidate] (9.237 ms) : 0, 9237
Flare Poller [baseline] (10.753 ms) : 0, 10753
Flare Poller [candidate] (11.104 ms) : 0, 11104
section iast
crashtracking [baseline] (1.466 ms) : 0, 1466
crashtracking [candidate] (1.459 ms) : 0, 1459
BytebuddyAgent [baseline] (820.593 ms) : 0, 820593
BytebuddyAgent [candidate] (816.247 ms) : 0, 816247
GlobalTracer [baseline] (234.52 ms) : 0, 234520
GlobalTracer [candidate] (232.227 ms) : 0, 232227
AppSec [baseline] (35.826 ms) : 0, 35826
AppSec [candidate] (35.107 ms) : 0, 35107
Debugger [baseline] (6.173 ms) : 0, 6173
Debugger [candidate] (6.141 ms) : 0, 6141
Remote Config [baseline] (632.52 µs) : 0, 633
Remote Config [candidate] (606.239 µs) : 0, 606
Telemetry [baseline] (8.696 ms) : 0, 8696
Telemetry [candidate] (8.461 ms) : 0, 8461
Flare Poller [baseline] (4.193 ms) : 0, 4193
Flare Poller [candidate] (4.164 ms) : 0, 4164
IAST [baseline] (27.349 ms) : 0, 27349
IAST [candidate] (26.858 ms) : 0, 26858

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.023 s) : 0, 1022669
Total [baseline] (10.735 s) : 0, 10734776
Agent [candidate] (1.02 s) : 0, 1020304
Total [candidate] (10.654 s) : 0, 10654023
section appsec
Agent [baseline] (1.194 s) : 0, 1194188
Total [baseline] (11.065 s) : 0, 11064767
Agent [candidate] (1.192 s) : 0, 1191747
Total [candidate] (10.922 s) : 0, 10921980
section iast
Agent [baseline] (1.161 s) : 0, 1160747
Total [baseline] (10.945 s) : 0, 10944780
Agent [candidate] (1.15 s) : 0, 1150035
Total [candidate] (10.874 s) : 0, 10873543
section profiling
Agent [baseline] (1.174 s) : 0, 1173930
Total [baseline] (11.139 s) : 0, 11139363
Agent [candidate] (1.165 s) : 0, 1164896
Total [candidate] (10.999 s) : 0, 10999251

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.023 s	-
Agent	appsec	1.194 s	171.519 ms (16.8%)
Agent	iast	1.161 s	138.078 ms (13.5%)
Agent	profiling	1.174 s	151.261 ms (14.8%)
Total	tracing	10.735 s	-
Total	appsec	11.065 s	329.99 ms (3.1%)
Total	iast	10.945 s	210.003 ms (2.0%)
Total	profiling	11.139 s	404.587 ms (3.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.02 s	-
Agent	appsec	1.192 s	171.442 ms (16.8%)
Agent	iast	1.15 s	129.73 ms (12.7%)
Agent	profiling	1.165 s	144.592 ms (14.2%)
Total	tracing	10.654 s	-
Total	appsec	10.922 s	267.957 ms (2.5%)
Total	iast	10.874 s	219.52 ms (2.1%)
Total	profiling	10.999 s	345.228 ms (3.2%)

gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.487 ms) : 0, 1487
crashtracking [candidate] (1.459 ms) : 0, 1459
BytebuddyAgent [baseline] (697.451 ms) : 0, 697451
BytebuddyAgent [candidate] (694.838 ms) : 0, 694838
GlobalTracer [baseline] (241.711 ms) : 0, 241711
GlobalTracer [candidate] (242.214 ms) : 0, 242214
AppSec [baseline] (32.779 ms) : 0, 32779
AppSec [candidate] (32.911 ms) : 0, 32911
Debugger [baseline] (6.429 ms) : 0, 6429
Debugger [candidate] (6.465 ms) : 0, 6465
Remote Config [baseline] (694.796 µs) : 0, 695
Remote Config [candidate] (683.706 µs) : 0, 684
Telemetry [baseline] (9.249 ms) : 0, 9249
Telemetry [candidate] (9.337 ms) : 0, 9337
Flare Poller [baseline] (11.787 ms) : 0, 11787
Flare Poller [candidate] (11.305 ms) : 0, 11305
section appsec
crashtracking [baseline] (1.464 ms) : 0, 1464
crashtracking [candidate] (1.466 ms) : 0, 1466
BytebuddyAgent [baseline] (717.61 ms) : 0, 717610
BytebuddyAgent [candidate] (716.003 ms) : 0, 716003
GlobalTracer [baseline] (234.487 ms) : 0, 234487
GlobalTracer [candidate] (233.867 ms) : 0, 233867
AppSec [baseline] (173.055 ms) : 0, 173055
AppSec [candidate] (174.463 ms) : 0, 174463
Debugger [baseline] (6.192 ms) : 0, 6192
Debugger [candidate] (6.19 ms) : 0, 6190
Remote Config [baseline] (645.382 µs) : 0, 645
Remote Config [candidate] (647.332 µs) : 0, 647
Telemetry [baseline] (10.071 ms) : 0, 10071
Telemetry [candidate] (9.361 ms) : 0, 9361
Flare Poller [baseline] (4.777 ms) : 0, 4777
Flare Poller [candidate] (4.024 ms) : 0, 4024
IAST [baseline] (24.843 ms) : 0, 24843
IAST [candidate] (24.703 ms) : 0, 24703
section iast
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.455 ms) : 0, 1455
BytebuddyAgent [baseline] (822.307 ms) : 0, 822307
BytebuddyAgent [candidate] (813.849 ms) : 0, 813849
GlobalTracer [baseline] (233.576 ms) : 0, 233576
GlobalTracer [candidate] (232.348 ms) : 0, 232348
AppSec [baseline] (35.614 ms) : 0, 35614
AppSec [candidate] (35.168 ms) : 0, 35168
Debugger [baseline] (6.078 ms) : 0, 6078
Debugger [candidate] (6.051 ms) : 0, 6051
Remote Config [baseline] (623.702 µs) : 0, 624
Remote Config [candidate] (599.493 µs) : 0, 599
Telemetry [baseline] (8.489 ms) : 0, 8489
Telemetry [candidate] (8.406 ms) : 0, 8406
Flare Poller [baseline] (4.205 ms) : 0, 4205
Flare Poller [candidate] (4.184 ms) : 0, 4184
IAST [baseline] (27.09 ms) : 0, 27090
IAST [candidate] (26.746 ms) : 0, 26746
section profiling
ProfilingAgent [baseline] (108.5 ms) : 0, 108500
ProfilingAgent [candidate] (107.505 ms) : 0, 107505
crashtracking [baseline] (1.433 ms) : 0, 1433
crashtracking [candidate] (1.434 ms) : 0, 1434
BytebuddyAgent [baseline] (729.115 ms) : 0, 729115
BytebuddyAgent [candidate] (724.948 ms) : 0, 724948
GlobalTracer [baseline] (219.894 ms) : 0, 219894
GlobalTracer [candidate] (217.145 ms) : 0, 217145
AppSec [baseline] (33.469 ms) : 0, 33469
AppSec [candidate] (33.101 ms) : 0, 33101
Debugger [baseline] (7.361 ms) : 0, 7361
Debugger [candidate] (6.461 ms) : 0, 6461
Remote Config [baseline] (720.244 µs) : 0, 720
Remote Config [candidate] (694.539 µs) : 0, 695
Telemetry [baseline] (15.978 ms) : 0, 15978
Telemetry [candidate] (15.823 ms) : 0, 15823
Flare Poller [baseline] (4.337 ms) : 0, 4337
Flare Poller [candidate] (4.947 ms) : 0, 4947
Profiling [baseline] (109.235 ms) : 0, 109235
Profiling [candidate] (108.13 ms) : 0, 108130

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	vandonr/fix
git_commit_date	1759752550	1759758855
git_commit_sha	`20dac8a`	`f99014d`
release_version	1.54.0-SNAPSHOT~20dac8a306	1.54.0-SNAPSHOT~f99014d8ad

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1759760715	1759760715
ci_job_id	1164768980	1164768980
ci_pipeline_id	78473758	78473758
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-gdm4iinw 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-gdm4iinw 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 6 performance improvements and 0 performance regressions! Performance is the same for 6 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:tracing:high_load	better [-630.730µs; -385.018µs] or [-7.675%; -4.685%]	unstable [-65.185op/s; +102.094op/s] or [-11.558%; +18.102%]	7.710ms	582.455op/s	8.218ms	564.000op/s
scenario:load:insecure-bank:profiling:high_load	better [-772.769µs; -451.143µs] or [-8.245%; -4.814%]	unstable [-59.538op/s; +95.521op/s] or [-12.021%; +19.286%]	8.760ms	513.273op/s	9.372ms	495.281op/s
scenario:load:petclinic:no_agent:high_load	better [-2.672ms; -2.028ms] or [-7.038%; -5.341%]	unstable [-0.588op/s; +16.813op/s] or [-0.478%; +13.655%]	35.617ms	131.238op/s	37.967ms	123.125op/s
scenario:load:petclinic:profiling:high_load	better [-2.216ms; -1.222ms] or [-4.620%; -2.548%]	unstable [-3.669op/s; +10.869op/s] or [-3.761%; +11.141%]	46.252ms	101.162op/s	47.972ms	97.562op/s
scenario:load:petclinic:code_origins:high_load	better [-1.678ms; -0.888ms] or [-3.843%; -2.034%]	unstable [-4.334op/s; +10.859op/s] or [-4.047%; +10.139%]	42.382ms	110.362op/s	43.665ms	107.100op/s
scenario:load:petclinic:iast:high_load	better [-2.781ms; -1.975ms] or [-6.120%; -4.346%]	unstable [-1.767op/s; +13.067op/s] or [-1.716%; +12.691%]	43.062ms	108.612op/s	45.440ms	102.963op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.493 ms) : 4438, 4548
.   : milestone, 4493,
iast (10.043 ms) : 9872, 10214
.   : milestone, 10043,
iast_FULL (13.943 ms) : 13666, 14220
.   : milestone, 13943,
iast_GLOBAL (11.129 ms) : 10931, 11328
.   : milestone, 11129,
profiling (9.372 ms) : 9214, 9530
.   : milestone, 9372,
tracing (8.218 ms) : 8098, 8338
.   : milestone, 8218,
section candidate
no_agent (4.395 ms) : 4341, 4449
.   : milestone, 4395,
iast (9.989 ms) : 9814, 10165
.   : milestone, 9989,
iast_FULL (14.448 ms) : 14159, 14738
.   : milestone, 14448,
iast_GLOBAL (11.208 ms) : 11000, 11416
.   : milestone, 11208,
profiling (8.76 ms) : 8620, 8900
.   : milestone, 8760,
tracing (7.71 ms) : 7602, 7819
.   : milestone, 7710,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.493 ms [4.438 ms, 4.548 ms]	-
iast	10.043 ms [9.872 ms, 10.214 ms]	5.55 ms (123.5%)
iast_FULL	13.943 ms [13.666 ms, 14.22 ms]	9.45 ms (210.4%)
iast_GLOBAL	11.129 ms [10.931 ms, 11.328 ms]	6.637 ms (147.7%)
profiling	9.372 ms [9.214 ms, 9.53 ms]	4.88 ms (108.6%)
tracing	8.218 ms [8.098 ms, 8.338 ms]	3.725 ms (82.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.395 ms [4.341 ms, 4.449 ms]	-
iast	9.989 ms [9.814 ms, 10.165 ms]	5.594 ms (127.3%)
iast_FULL	14.448 ms [14.159 ms, 14.738 ms]	10.053 ms (228.7%)
iast_GLOBAL	11.208 ms [11.0 ms, 11.416 ms]	6.813 ms (155.0%)
profiling	8.76 ms [8.62 ms, 8.9 ms]	4.365 ms (99.3%)
tracing	7.71 ms [7.602 ms, 7.819 ms]	3.315 ms (75.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.967 ms) : 37652, 38283
.   : milestone, 37967,
appsec (46.738 ms) : 46325, 47152
.   : milestone, 46738,
code_origins (43.665 ms) : 43298, 44032
.   : milestone, 43665,
iast (45.44 ms) : 45058, 45822
.   : milestone, 45440,
profiling (47.972 ms) : 47485, 48458
.   : milestone, 47972,
tracing (45.777 ms) : 45391, 46164
.   : milestone, 45777,
section candidate
no_agent (35.617 ms) : 35335, 35899
.   : milestone, 35617,
appsec (47.779 ms) : 47368, 48191
.   : milestone, 47779,
code_origins (42.382 ms) : 42015, 42749
.   : milestone, 42382,
iast (43.062 ms) : 42695, 43429
.   : milestone, 43062,
profiling (46.252 ms) : 45817, 46688
.   : milestone, 46252,
tracing (45.53 ms) : 45130, 45930
.   : milestone, 45530,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.967 ms [37.652 ms, 38.283 ms]	-
appsec	46.738 ms [46.325 ms, 47.152 ms]	8.771 ms (23.1%)
code_origins	43.665 ms [43.298 ms, 44.032 ms]	5.698 ms (15.0%)
iast	45.44 ms [45.058 ms, 45.822 ms]	7.472 ms (19.7%)
profiling	47.972 ms [47.485 ms, 48.458 ms]	10.004 ms (26.3%)
tracing	45.777 ms [45.391 ms, 46.164 ms]	7.81 ms (20.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.617 ms [35.335 ms, 35.899 ms]	-
appsec	47.779 ms [47.368 ms, 48.191 ms]	12.162 ms (34.1%)
code_origins	42.382 ms [42.015 ms, 42.749 ms]	6.765 ms (19.0%)
iast	43.062 ms [42.695 ms, 43.429 ms]	7.445 ms (20.9%)
profiling	46.252 ms [45.817 ms, 46.688 ms]	10.635 ms (29.9%)
tracing	45.53 ms [45.13 ms, 45.93 ms]	9.913 ms (27.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	vandonr/fix
git_commit_date	1759752550	1759758855
git_commit_sha	`20dac8a`	`f99014d`
release_version	1.54.0-SNAPSHOT~20dac8a306	1.54.0-SNAPSHOT~f99014d8ad

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1759761165	1759761165
ci_job_id	1164768981	1164768981
ci_pipeline_id	78473758	78473758
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-7tqofhes 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-7tqofhes 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.5 s) : 15500000, 15500000
.   : milestone, 15500000,
appsec (15.193 s) : 15193000, 15193000
.   : milestone, 15193000,
iast (18.69 s) : 18690000, 18690000
.   : milestone, 18690000,
iast_GLOBAL (18.182 s) : 18182000, 18182000
.   : milestone, 18182000,
profiling (15.458 s) : 15458000, 15458000
.   : milestone, 15458000,
tracing (15.077 s) : 15077000, 15077000
.   : milestone, 15077000,
section candidate
no_agent (15.65 s) : 15650000, 15650000
.   : milestone, 15650000,
appsec (15.039 s) : 15039000, 15039000
.   : milestone, 15039000,
iast (18.421 s) : 18421000, 18421000
.   : milestone, 18421000,
iast_GLOBAL (17.634 s) : 17634000, 17634000
.   : milestone, 17634000,
profiling (15.422 s) : 15422000, 15422000
.   : milestone, 15422000,
tracing (15.207 s) : 15207000, 15207000
.   : milestone, 15207000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.5 s [15.5 s, 15.5 s]	-
appsec	15.193 s [15.193 s, 15.193 s]	-307.0 ms (-2.0%)
iast	18.69 s [18.69 s, 18.69 s]	3.19 s (20.6%)
iast_GLOBAL	18.182 s [18.182 s, 18.182 s]	2.682 s (17.3%)
profiling	15.458 s [15.458 s, 15.458 s]	-42.0 ms (-0.3%)
tracing	15.077 s [15.077 s, 15.077 s]	-423.0 ms (-2.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.65 s [15.65 s, 15.65 s]	-
appsec	15.039 s [15.039 s, 15.039 s]	-611.0 ms (-3.9%)
iast	18.421 s [18.421 s, 18.421 s]	2.771 s (17.7%)
iast_GLOBAL	17.634 s [17.634 s, 17.634 s]	1.984 s (12.7%)
profiling	15.422 s [15.422 s, 15.422 s]	-228.0 ms (-1.5%)
tracing	15.207 s [15.207 s, 15.207 s]	-443.0 ms (-2.8%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~f99014d8ad, baseline=1.54.0-SNAPSHOT~20dac8a306
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.475 ms) : 1463, 1486
.   : milestone, 1475,
appsec (3.724 ms) : 3504, 3945
.   : milestone, 3724,
iast (2.207 ms) : 2144, 2270
.   : milestone, 2207,
iast_GLOBAL (2.253 ms) : 2189, 2317
.   : milestone, 2253,
profiling (2.04 ms) : 1989, 2091
.   : milestone, 2040,
tracing (2.026 ms) : 1977, 2075
.   : milestone, 2026,
section candidate
no_agent (1.479 ms) : 1467, 1490
.   : milestone, 1479,
appsec (3.74 ms) : 3522, 3959
.   : milestone, 3740,
iast (2.199 ms) : 2136, 2262
.   : milestone, 2199,
iast_GLOBAL (2.252 ms) : 2188, 2315
.   : milestone, 2252,
profiling (2.047 ms) : 1996, 2098
.   : milestone, 2047,
tracing (2.036 ms) : 1987, 2086
.   : milestone, 2036,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.463 ms, 1.486 ms]	-
appsec	3.724 ms [3.504 ms, 3.945 ms]	2.249 ms (152.5%)
iast	2.207 ms [2.144 ms, 2.27 ms]	732.559 µs (49.7%)
iast_GLOBAL	2.253 ms [2.189 ms, 2.317 ms]	778.508 µs (52.8%)
profiling	2.04 ms [1.989 ms, 2.091 ms]	565.176 µs (38.3%)
tracing	2.026 ms [1.977 ms, 2.075 ms]	551.795 µs (37.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.467 ms, 1.49 ms]	-
appsec	3.74 ms [3.522 ms, 3.959 ms]	2.262 ms (153.0%)
iast	2.199 ms [2.136 ms, 2.262 ms]	720.609 µs (48.7%)
iast_GLOBAL	2.252 ms [2.188 ms, 2.315 ms]	773.025 µs (52.3%)
profiling	2.047 ms [1.996 ms, 2.098 ms]	568.086 µs (38.4%)
tracing	2.036 ms [1.987 ms, 2.086 ms]	557.872 µs (37.7%)

try to fix pekko

f99014d

vandonr added type: bug Bug report and fix inst: others All other instrumentations tag: flaky test Flaky tests labels Oct 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix pekko instrumentation in a rare edge case #9676

Fix pekko instrumentation in a rare edge case #9676

Uh oh!

vandonr commented Oct 6, 2025

Uh oh!

pr-commenter bot commented Oct 6, 2025

Uh oh!

Uh oh!

	// Since we haven't instrumented the pekko stream state machine, we can't rely
	// on spans and scopes being propagated during the push and pull of the
	// element. Instead we let the scope leak intentionally here and clean it
	// up when the user response comes back, or in the actor message processing
	// instrumentation that drives this state machine.

Fix pekko instrumentation in a rare edge case #9676

Are you sure you want to change the base?

Fix pekko instrumentation in a rare edge case #9676

Uh oh!

Conversation

vandonr commented Oct 6, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Oct 6, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!