Mark AppSecActivationSmokeTest as flaky in oracle8 #10037

manuel-alvarez-alvarez · 2025-11-26T14:49:33Z

What Does This Do

Marks the AppSecActivationSmokeTest as flaky in oracle8
Added support for using closures in @Flaky annotations, alongside Predicate implementations, to simplify flaky-test filtering.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

utils/test-utils/src/main/groovy/datadog/trace/test/util/Predicates.java

AlexeyKuznetsov-DD

LGTM in general, left minor comment.
Maybe we do not need this new class and can use methods from JavaVirtualMachine ?

manuel-alvarez-alvarez · 2025-11-26T15:33:58Z

LGTM in general, left minor comment. Maybe we do not need this new class and can use methods from JavaVirtualMachine ?

Flaky test management is built in Java so we don't have access to groovy stuff in there (the condition just points to a class that implements Predicate), we could migrate it to groovy and use Closure there.

pr-commenter · 2025-11-26T16:19:49Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	malvarez/appsec-activation-flaky-oracle8
git_commit_date	1765268180	1765269178
git_commit_sha	`2229c62`	`f7601e7`
release_version	1.57.0-SNAPSHOT~2229c621bd	1.57.0-SNAPSHOT~f7601e7c51

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1765271140	1765271140
ci_job_id	1278448836	1278448836
ci_pipeline_id	85500949	85500949
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-lski0j7l 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-lski0j7l 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 5 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.095 s) : 0, 1094610
Total [baseline] (10.998 s) : 0, 10997530
Agent [candidate] (1.086 s) : 0, 1085868
Total [candidate] (10.874 s) : 0, 10874237
section appsec
Agent [baseline] (1.267 s) : 0, 1266890
Total [baseline] (11.116 s) : 0, 11115799
Agent [candidate] (1.265 s) : 0, 1264885
Total [candidate] (11.113 s) : 0, 11112924
section iast
Agent [baseline] (1.227 s) : 0, 1226612
Total [baseline] (11.213 s) : 0, 11213050
Agent [candidate] (1.227 s) : 0, 1227230
Total [candidate] (11.251 s) : 0, 11250561
section profiling
Agent [baseline] (1.208 s) : 0, 1207921
Total [baseline] (11.016 s) : 0, 11016461
Agent [candidate] (1.207 s) : 0, 1206524
Total [candidate] (11.023 s) : 0, 11023414

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.095 s	-
Agent	appsec	1.267 s	172.28 ms (15.7%)
Agent	iast	1.227 s	132.002 ms (12.1%)
Agent	profiling	1.208 s	113.311 ms (10.4%)
Total	tracing	10.998 s	-
Total	appsec	11.116 s	118.269 ms (1.1%)
Total	iast	11.213 s	215.521 ms (2.0%)
Total	profiling	11.016 s	18.932 ms (0.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.086 s	-
Agent	appsec	1.265 s	179.017 ms (16.5%)
Agent	iast	1.227 s	141.361 ms (13.0%)
Agent	profiling	1.207 s	120.655 ms (11.1%)
Total	tracing	10.874 s	-
Total	appsec	11.113 s	238.687 ms (2.2%)
Total	iast	11.251 s	376.324 ms (3.5%)
Total	profiling	11.023 s	149.178 ms (1.4%)

gantt
    title petclinic - break down per module: candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.224 ms) : 0, 1224
crashtracking [candidate] (1.199 ms) : 0, 1199
BytebuddyAgent [baseline] (657.996 ms) : 0, 657996
BytebuddyAgent [candidate] (650.588 ms) : 0, 650588
GlobalTracer [baseline] (284.108 ms) : 0, 284108
GlobalTracer [candidate] (283.688 ms) : 0, 283688
AppSec [baseline] (32.849 ms) : 0, 32849
AppSec [candidate] (32.822 ms) : 0, 32822
Debugger [baseline] (68.216 ms) : 0, 68216
Debugger [candidate] (68.502 ms) : 0, 68502
Remote Config [baseline] (662.902 µs) : 0, 663
Remote Config [candidate] (642.17 µs) : 0, 642
Telemetry [baseline] (9.823 ms) : 0, 9823
Telemetry [candidate] (9.037 ms) : 0, 9037
Flare Poller [baseline] (3.807 ms) : 0, 3807
Flare Poller [candidate] (3.829 ms) : 0, 3829
section appsec
crashtracking [baseline] (1.201 ms) : 0, 1201
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (690.572 ms) : 0, 690572
BytebuddyAgent [candidate] (688.749 ms) : 0, 688749
GlobalTracer [baseline] (259.933 ms) : 0, 259933
GlobalTracer [candidate] (259.581 ms) : 0, 259581
AppSec [baseline] (174.26 ms) : 0, 174260
AppSec [candidate] (175.185 ms) : 0, 175185
Debugger [baseline] (67.036 ms) : 0, 67036
Debugger [candidate] (66.307 ms) : 0, 66307
Remote Config [baseline] (703.412 µs) : 0, 703
Remote Config [candidate] (702.788 µs) : 0, 703
Telemetry [baseline] (9.149 ms) : 0, 9149
Telemetry [candidate] (9.144 ms) : 0, 9144
Flare Poller [baseline] (3.894 ms) : 0, 3894
Flare Poller [candidate] (3.978 ms) : 0, 3978
IAST [baseline] (24.549 ms) : 0, 24549
IAST [candidate] (24.502 ms) : 0, 24502
section iast
crashtracking [baseline] (1.205 ms) : 0, 1205
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (791.699 ms) : 0, 791699
BytebuddyAgent [candidate] (792.914 ms) : 0, 792914
GlobalTracer [baseline] (256.361 ms) : 0, 256361
GlobalTracer [candidate] (255.892 ms) : 0, 255892
AppSec [baseline] (35.471 ms) : 0, 35471
AppSec [candidate] (35.881 ms) : 0, 35881
Debugger [baseline] (66.527 ms) : 0, 66527
Debugger [candidate] (66.112 ms) : 0, 66112
Remote Config [baseline] (565.64 µs) : 0, 566
Remote Config [candidate] (593.288 µs) : 0, 593
Telemetry [baseline] (8.525 ms) : 0, 8525
Telemetry [candidate] (8.487 ms) : 0, 8487
Flare Poller [baseline] (3.505 ms) : 0, 3505
Flare Poller [candidate] (3.481 ms) : 0, 3481
IAST [baseline] (27.302 ms) : 0, 27302
IAST [candidate] (27.203 ms) : 0, 27203
section profiling
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (702.706 ms) : 0, 702706
BytebuddyAgent [candidate] (703.692 ms) : 0, 703692
GlobalTracer [baseline] (221.056 ms) : 0, 221056
GlobalTracer [candidate] (221.441 ms) : 0, 221441
AppSec [baseline] (32.244 ms) : 0, 32244
AppSec [candidate] (32.215 ms) : 0, 32215
Debugger [baseline] (69.15 ms) : 0, 69150
Debugger [candidate] (67.704 ms) : 0, 67704
Remote Config [baseline] (610.267 µs) : 0, 610
Remote Config [candidate] (629.231 µs) : 0, 629
Telemetry [baseline] (9.008 ms) : 0, 9008
Telemetry [candidate] (9.043 ms) : 0, 9043
Flare Poller [baseline] (3.806 ms) : 0, 3806
Flare Poller [candidate] (3.804 ms) : 0, 3804
ProfilingAgent [baseline] (98.46 ms) : 0, 98460
ProfilingAgent [candidate] (97.131 ms) : 0, 97131
Profiling [baseline] (99.038 ms) : 0, 99038
Profiling [candidate] (97.699 ms) : 0, 97699

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.085 s) : 0, 1085299
Total [baseline] (8.756 s) : 0, 8755813
Agent [candidate] (1.091 s) : 0, 1091206
Total [candidate] (8.773 s) : 0, 8773174
section iast
Agent [baseline] (1.224 s) : 0, 1224449
Total [baseline] (9.472 s) : 0, 9471803
Agent [candidate] (1.233 s) : 0, 1232623
Total [candidate] (9.538 s) : 0, 9537930

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.085 s	-
Agent	iast	1.224 s	139.15 ms (12.8%)
Total	tracing	8.756 s	-
Total	iast	9.472 s	715.99 ms (8.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.091 s	-
Agent	iast	1.233 s	141.417 ms (13.0%)
Total	tracing	8.773 s	-
Total	iast	9.538 s	764.756 ms (8.7%)

gantt
    title insecure-bank - break down per module: candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.216 ms) : 0, 1216
crashtracking [candidate] (1.196 ms) : 0, 1196
BytebuddyAgent [baseline] (653.131 ms) : 0, 653131
BytebuddyAgent [candidate] (655.272 ms) : 0, 655272
GlobalTracer [baseline] (281.974 ms) : 0, 281974
GlobalTracer [candidate] (284.606 ms) : 0, 284606
AppSec [baseline] (32.352 ms) : 0, 32352
AppSec [candidate] (32.729 ms) : 0, 32729
Debugger [baseline] (67.354 ms) : 0, 67354
Debugger [candidate] (68.008 ms) : 0, 68008
Remote Config [baseline] (650.521 µs) : 0, 651
Remote Config [candidate] (655.233 µs) : 0, 655
Telemetry [baseline] (9.094 ms) : 0, 9094
Telemetry [candidate] (9.119 ms) : 0, 9119
Flare Poller [baseline] (3.769 ms) : 0, 3769
Flare Poller [candidate] (3.812 ms) : 0, 3812
section iast
crashtracking [baseline] (1.199 ms) : 0, 1199
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (792.848 ms) : 0, 792848
BytebuddyAgent [candidate] (797.469 ms) : 0, 797469
GlobalTracer [baseline] (255.634 ms) : 0, 255634
GlobalTracer [candidate] (257.601 ms) : 0, 257601
AppSec [baseline] (35.625 ms) : 0, 35625
AppSec [candidate] (35.732 ms) : 0, 35732
Debugger [baseline] (64.353 ms) : 0, 64353
Debugger [candidate] (65.215 ms) : 0, 65215
Remote Config [baseline] (566.617 µs) : 0, 567
Remote Config [candidate] (569.215 µs) : 0, 569
Telemetry [baseline] (8.451 ms) : 0, 8451
Telemetry [candidate] (8.459 ms) : 0, 8459
Flare Poller [baseline] (3.457 ms) : 0, 3457
Flare Poller [candidate] (3.503 ms) : 0, 3503
IAST [baseline] (26.841 ms) : 0, 26841
IAST [candidate] (27.287 ms) : 0, 27287

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	malvarez/appsec-activation-flaky-oracle8
git_commit_date	1765268180	1765269178
git_commit_sha	`2229c62`	`f7601e7`
release_version	1.57.0-SNAPSHOT~2229c621bd	1.57.0-SNAPSHOT~f7601e7c51

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1765271623	1765271623
ci_job_id	1278448839	1278448839
ci_pipeline_id	85500949	85500949
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-09y3wmd8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-09y3wmd8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 4 performance regressions! Performance is the same for 14 metrics, 18 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:iast_GLOBAL:high_load	worse [+91.554µs; +226.109µs] or [+3.358%; +8.293%]	unsure [+106.266µs; +554.043µs] or [+1.359%; +7.087%]	unstable [-195.369op/s; +86.619op/s] or [-14.948%; +6.627%]	2.885ms	8.148ms	1252.594op/s	2.726ms	7.818ms	1306.969op/s
scenario:load:insecure-bank:iast_FULL:high_load	worse [+381.610µs; +717.126µs] or [+7.715%; +14.497%]	unstable [+0.675ms; +1.932ms] or [+5.840%; +16.720%]	unstable [-127.089op/s; +65.714op/s] or [-16.136%; +8.343%]	5.496ms	12.859ms	756.938op/s	4.947ms	11.556ms	787.625op/s
scenario:load:petclinic:code_origins:high_load	worse [+1.826ms; +2.266ms] or [+10.494%; +13.020%]	worse [+2.144ms; +3.206ms] or [+7.595%; +11.357%]	unstable [-55.613op/s; -0.387op/s] or [-21.056%; -0.147%]	19.447ms	30.901ms	236.125op/s	17.401ms	28.226ms	264.125op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.221 ms) : 1209, 1233
.   : milestone, 1221,
iast (3.293 ms) : 3248, 3339
.   : milestone, 3293,
iast_FULL (5.671 ms) : 5614, 5729
.   : milestone, 5671,
iast_GLOBAL (3.506 ms) : 3450, 3563
.   : milestone, 3506,
profiling (1.967 ms) : 1950, 1984
.   : milestone, 1967,
tracing (1.816 ms) : 1800, 1832
.   : milestone, 1816,
section candidate
no_agent (1.218 ms) : 1206, 1230
.   : milestone, 1218,
iast (3.27 ms) : 3229, 3311
.   : milestone, 3270,
iast_FULL (6.113 ms) : 6050, 6175
.   : milestone, 6113,
iast_GLOBAL (3.663 ms) : 3608, 3717
.   : milestone, 3663,
profiling (2.056 ms) : 2038, 2073
.   : milestone, 2056,
tracing (1.807 ms) : 1792, 1822
.   : milestone, 1807,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.221 ms [1.209 ms, 1.233 ms]	-
iast	3.293 ms [3.248 ms, 3.339 ms]	2.073 ms (169.8%)
iast_FULL	5.671 ms [5.614 ms, 5.729 ms]	4.451 ms (364.6%)
iast_GLOBAL	3.506 ms [3.45 ms, 3.563 ms]	2.285 ms (187.2%)
profiling	1.967 ms [1.95 ms, 1.984 ms]	745.98 µs (61.1%)
tracing	1.816 ms [1.8 ms, 1.832 ms]	595.295 µs (48.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.218 ms [1.206 ms, 1.23 ms]	-
iast	3.27 ms [3.229 ms, 3.311 ms]	2.052 ms (168.5%)
iast_FULL	6.113 ms [6.05 ms, 6.175 ms]	4.895 ms (401.9%)
iast_GLOBAL	3.663 ms [3.608 ms, 3.717 ms]	2.445 ms (200.7%)
profiling	2.056 ms [2.038 ms, 2.073 ms]	837.614 µs (68.8%)
tracing	1.807 ms [1.792 ms, 1.822 ms]	589.211 µs (48.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.245 ms) : 17073, 17416
.   : milestone, 17245,
appsec (19.891 ms) : 19687, 20095
.   : milestone, 19891,
code_origins (17.663 ms) : 17490, 17837
.   : milestone, 17663,
iast (17.887 ms) : 17705, 18068
.   : milestone, 17887,
profiling (19.644 ms) : 19446, 19842
.   : milestone, 19644,
tracing (17.743 ms) : 17565, 17920
.   : milestone, 17743,
section candidate
no_agent (18.146 ms) : 17957, 18335
.   : milestone, 18146,
appsec (19.539 ms) : 19338, 19740
.   : milestone, 19539,
code_origins (19.769 ms) : 19572, 19965
.   : milestone, 19769,
iast (17.726 ms) : 17548, 17903
.   : milestone, 17726,
profiling (19.736 ms) : 19536, 19937
.   : milestone, 19736,
tracing (17.807 ms) : 17628, 17986
.   : milestone, 17807,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.245 ms [17.073 ms, 17.416 ms]	-
appsec	19.891 ms [19.687 ms, 20.095 ms]	2.646 ms (15.3%)
code_origins	17.663 ms [17.49 ms, 17.837 ms]	418.657 µs (2.4%)
iast	17.887 ms [17.705 ms, 18.068 ms]	641.814 µs (3.7%)
profiling	19.644 ms [19.446 ms, 19.842 ms]	2.399 ms (13.9%)
tracing	17.743 ms [17.565 ms, 17.92 ms]	497.824 µs (2.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.146 ms [17.957 ms, 18.335 ms]	-
appsec	19.539 ms [19.338 ms, 19.74 ms]	1.393 ms (7.7%)
code_origins	19.769 ms [19.572 ms, 19.965 ms]	1.623 ms (8.9%)
iast	17.726 ms [17.548 ms, 17.903 ms]	-420.104 µs (-2.3%)
profiling	19.736 ms [19.536 ms, 19.937 ms]	1.591 ms (8.8%)
tracing	17.807 ms [17.628 ms, 17.986 ms]	-339.188 µs (-1.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	malvarez/appsec-activation-flaky-oracle8
git_commit_date	1765268180	1765269178
git_commit_sha	`2229c62`	`f7601e7`
release_version	1.57.0-SNAPSHOT~2229c621bd	1.57.0-SNAPSHOT~f7601e7c51

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1765271257	1765271257
ci_job_id	1278448842	1278448842
ci_pipeline_id	85500949	85500949
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-0algxtkl 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-0algxtkl 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.581 s) : 15581000, 15581000
.   : milestone, 15581000,
appsec (14.562 s) : 14562000, 14562000
.   : milestone, 14562000,
iast (18.809 s) : 18809000, 18809000
.   : milestone, 18809000,
iast_GLOBAL (18.303 s) : 18303000, 18303000
.   : milestone, 18303000,
profiling (14.672 s) : 14672000, 14672000
.   : milestone, 14672000,
tracing (14.625 s) : 14625000, 14625000
.   : milestone, 14625000,
section candidate
no_agent (15.571 s) : 15571000, 15571000
.   : milestone, 15571000,
appsec (14.499 s) : 14499000, 14499000
.   : milestone, 14499000,
iast (18.247 s) : 18247000, 18247000
.   : milestone, 18247000,
iast_GLOBAL (18.099 s) : 18099000, 18099000
.   : milestone, 18099000,
profiling (15.15 s) : 15150000, 15150000
.   : milestone, 15150000,
tracing (14.9 s) : 14900000, 14900000
.   : milestone, 14900000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.581 s [15.581 s, 15.581 s]	-
appsec	14.562 s [14.562 s, 14.562 s]	-1.019 s (-6.5%)
iast	18.809 s [18.809 s, 18.809 s]	3.228 s (20.7%)
iast_GLOBAL	18.303 s [18.303 s, 18.303 s]	2.722 s (17.5%)
profiling	14.672 s [14.672 s, 14.672 s]	-909.0 ms (-5.8%)
tracing	14.625 s [14.625 s, 14.625 s]	-956.0 ms (-6.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.571 s [15.571 s, 15.571 s]	-
appsec	14.499 s [14.499 s, 14.499 s]	-1.072 s (-6.9%)
iast	18.247 s [18.247 s, 18.247 s]	2.676 s (17.2%)
iast_GLOBAL	18.099 s [18.099 s, 18.099 s]	2.528 s (16.2%)
profiling	15.15 s [15.15 s, 15.15 s]	-421.0 ms (-2.7%)
tracing	14.9 s [14.9 s, 14.9 s]	-671.0 ms (-4.3%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~f7601e7c51, baseline=1.57.0-SNAPSHOT~2229c621bd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.479 ms) : 1467, 1490
.   : milestone, 1479,
appsec (3.677 ms) : 3461, 3893
.   : milestone, 3677,
iast (2.23 ms) : 2165, 2295
.   : milestone, 2230,
iast_GLOBAL (2.272 ms) : 2207, 2338
.   : milestone, 2272,
profiling (2.067 ms) : 2015, 2120
.   : milestone, 2067,
tracing (2.057 ms) : 2006, 2108
.   : milestone, 2057,
section candidate
no_agent (1.48 ms) : 1468, 1491
.   : milestone, 1480,
appsec (3.679 ms) : 3462, 3897
.   : milestone, 3679,
iast (2.23 ms) : 2164, 2295
.   : milestone, 2230,
iast_GLOBAL (2.275 ms) : 2209, 2340
.   : milestone, 2275,
profiling (2.09 ms) : 2035, 2144
.   : milestone, 2090,
tracing (2.05 ms) : 1999, 2102
.   : milestone, 2050,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.467 ms, 1.49 ms]	-
appsec	3.677 ms [3.461 ms, 3.893 ms]	2.198 ms (148.7%)
iast	2.23 ms [2.165 ms, 2.295 ms]	751.446 µs (50.8%)
iast_GLOBAL	2.272 ms [2.207 ms, 2.338 ms]	793.887 µs (53.7%)
profiling	2.067 ms [2.015 ms, 2.12 ms]	588.771 µs (39.8%)
tracing	2.057 ms [2.006 ms, 2.108 ms]	578.426 µs (39.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.48 ms [1.468 ms, 1.491 ms]	-
appsec	3.679 ms [3.462 ms, 3.897 ms]	2.2 ms (148.6%)
iast	2.23 ms [2.164 ms, 2.295 ms]	749.77 µs (50.7%)
iast_GLOBAL	2.275 ms [2.209 ms, 2.34 ms]	794.79 µs (53.7%)
profiling	2.09 ms [2.035 ms, 2.144 ms]	609.724 µs (41.2%)
tracing	2.05 ms [1.999 ms, 2.102 ms]	570.568 µs (38.6%)

AlexeyKuznetsov-DD · 2025-11-26T18:56:26Z

LGTM in general, left minor comment. Maybe we do not need this new class and can use methods from JavaVirtualMachine ?

Flaky test management is built in Java so we don't have access to groovy stuff in there (the condition just points to a class that implements Predicate), we could migrate it to groovy and use Closure there.

Not sure that if my understanding correct, but I can see in our code base code in groovy like this:

@IgnoreIf(reason = "JVM crash with OpenJ9", value = {
  JavaVirtualMachine.isJ9()
})
class ChannelContextTest extends Specification {

Does it make sense? Or it is because @Flaky needs Predicate?

AlexeyKuznetsov-DD · 2025-11-26T19:03:45Z

Does it make sense? Or it is because @Flaky needs Predicate?

Just checked, this should work:
@Flaky(condition = () -> JavaVirtualMachine.isOracleJDK8())

utils/test-utils/src/main/groovy/datadog/trace/test/util/Predicates.java

manuel-alvarez-alvarez · 2025-11-27T10:40:02Z

LGTM in general, left minor comment. Maybe we do not need this new class and can use methods from JavaVirtualMachine ?

Flaky test management is built in Java so we don't have access to groovy stuff in there (the condition just points to a class that implements Predicate), we could migrate it to groovy and use Closure there.

Not sure that if my understanding correct, but I can see in our code base code in groovy like this:
@IgnoreIf(reason = "JVM crash with OpenJ9", value = {
  JavaVirtualMachine.isJ9()
})
class ChannelContextTest extends Specification {
Does it make sense? Or it is because @Flaky needs Predicate?

That's what I mentioned before, @Flaky is built in java so it receives a Predicate, when you add a Closure in groovy you end up with something like:

    @Flaky(
        condition = __spock_feature_2_0_closure2.class,
        value = "Telemetry product change event flakes in oracle8"
    )

Which is no longer a Predicate and our code fails to handle it, now I tweaked it so it should be able to deal with Closure as well.

PS: IgnoreIf works as it's groovy and receives a closure

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD})
@ExtensionAnnotation(IgnoreIfExtension.class)
@Repeatable(Container.class)
public @interface IgnoreIf {
    Class<? extends Closure> value();

    @Retention(RetentionPolicy.RUNTIME)
    @Target({ElementType.TYPE, ElementType.METHOD})
    public @interface Container {
        IgnoreIf[] value();
    }
}

PerfectSlayer

Left a minor request before merging.
Looking good otherwise 👍

components/environment/src/main/java/datadog/environment/JavaVirtualMachine.java

Flacky · 2025-12-02T10:17:16Z

Looks like my github tag as accidentally been added into the PR description, could it be corrected please? 🙂

PerfectSlayer · 2025-12-03T09:44:11Z

Sorry, this was a misspelled Java annotation (they start with an @ as the GitHub user mention system 😬 ). I just fix the typo

…ivation-flaky-oracle8

manuel-alvarez-alvarez requested a review from a team as a code owner November 26, 2025 14:49

manuel-alvarez-alvarez added the comp: testing Testing label Nov 26, 2025

manuel-alvarez-alvarez requested a review from mcculls November 26, 2025 14:49

manuel-alvarez-alvarez added the tag: no release notes Changes to exclude from release notes label Nov 26, 2025

AlexeyKuznetsov-DD reviewed Nov 26, 2025

View reviewed changes

utils/test-utils/src/main/groovy/datadog/trace/test/util/Predicates.java Outdated Show resolved Hide resolved

AlexeyKuznetsov-DD reviewed Nov 26, 2025

View reviewed changes

AlexeyKuznetsov-DD approved these changes Nov 26, 2025

View reviewed changes

AlexeyKuznetsov-DD reviewed Nov 26, 2025

View reviewed changes

utils/test-utils/src/main/groovy/datadog/trace/test/util/Predicates.java Outdated Show resolved Hide resolved

PerfectSlayer added tag: flaky test Flaky tests tag: flaky test/disabled Disabled flaky tests comp: asm waf Application Security Management (WAF) and removed comp: testing Testing labels Nov 27, 2025

manuel-alvarez-alvarez added 3 commits November 27, 2025 11:36

Mark AppSecActivationSmokeTest as flaky in oracle8

6d50bdb

Remove System.getProperty from predicates

53849dd

Get rid of the predicates class

c34ff34

manuel-alvarez-alvarez force-pushed the malvarez/appsec-activation-flaky-oracle8 branch from 929aaf3 to c34ff34 Compare November 27, 2025 10:36

manuel-alvarez-alvarez requested review from a team as code owners November 27, 2025 10:36

manuel-alvarez-alvarez requested review from claponcet, jandro996 and sarahchen6 and removed request for a team November 27, 2025 10:36

manuel-alvarez-alvarez requested a review from AlexeyKuznetsov-DD November 27, 2025 10:43

sarahchen6 approved these changes Dec 1, 2025

View reviewed changes

AlexeyKuznetsov-DD approved these changes Dec 1, 2025

View reviewed changes

AlexeyKuznetsov-DD requested a review from PerfectSlayer December 1, 2025 18:11

PerfectSlayer reviewed Dec 2, 2025

View reviewed changes

components/environment/src/main/java/datadog/environment/JavaVirtualMachine.java Show resolved Hide resolved

manuel-alvarez-alvarez added 3 commits December 5, 2025 15:39

Merge remote-tracking branch 'origin/master' into malvarez/appsec-act…

083aa8e

…ivation-flaky-oracle8

Merge remote-tracking branch 'origin/master' into malvarez/appsec-act…

ea2cd24

…ivation-flaky-oracle8

Add test to JavaVirtualMachineTest

f7601e7

manuel-alvarez-alvarez temporarily deployed to ci-build December 9, 2025 08:35 — with GitHub Actions Inactive

manuel-alvarez-alvarez merged commit 9885025 into master Dec 9, 2025
539 checks passed

manuel-alvarez-alvarez deleted the malvarez/appsec-activation-flaky-oracle8 branch December 9, 2025 09:56

github-actions bot added this to the 1.57.0 milestone Dec 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Mark AppSecActivationSmokeTest as flaky in oracle8 #10037

Mark AppSecActivationSmokeTest as flaky in oracle8 #10037

Uh oh!

manuel-alvarez-alvarez commented Nov 26, 2025 •

edited by PerfectSlayer

Loading

Uh oh!

Uh oh!

AlexeyKuznetsov-DD left a comment

Uh oh!

manuel-alvarez-alvarez commented Nov 26, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Nov 26, 2025 •

edited

Loading

Uh oh!

AlexeyKuznetsov-DD commented Nov 26, 2025

Uh oh!

AlexeyKuznetsov-DD commented Nov 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

manuel-alvarez-alvarez commented Nov 27, 2025 •

edited

Loading

Uh oh!

PerfectSlayer left a comment

Uh oh!

Uh oh!

Flacky commented Dec 2, 2025

Uh oh!

PerfectSlayer commented Dec 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Mark AppSecActivationSmokeTest as flaky in oracle8 #10037

Mark AppSecActivationSmokeTest as flaky in oracle8 #10037

Uh oh!

Conversation

manuel-alvarez-alvarez commented Nov 26, 2025 • edited by PerfectSlayer Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

Uh oh!

AlexeyKuznetsov-DD left a comment

Choose a reason for hiding this comment

Uh oh!

manuel-alvarez-alvarez commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

AlexeyKuznetsov-DD commented Nov 26, 2025

Uh oh!

AlexeyKuznetsov-DD commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

manuel-alvarez-alvarez commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Flacky commented Dec 2, 2025

Uh oh!

PerfectSlayer commented Dec 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

manuel-alvarez-alvarez commented Nov 26, 2025 •

edited by PerfectSlayer

Loading

manuel-alvarez-alvarez commented Nov 26, 2025 •

edited

Loading

pr-commenter bot commented Nov 26, 2025 •

edited

Loading

AlexeyKuznetsov-DD commented Nov 26, 2025 •

edited

Loading

manuel-alvarez-alvarez commented Nov 27, 2025 •

edited

Loading