UBI10 runtime and builder #386

Karm · 2025-11-27T23:16:09Z

Inspired by @gsmet 's commit

74d0fbe

from PR #384

This PR:

moves runtime UBI10 images from 10.0 to 10.1
adds a builder x runtime compatibility matrix BUILDER_RUNTIME_COMPATIBILITY
keeps a bit of logic regarding dev images in isBuilderImageIncompatible

Why retain an ordered list of those runtime images even if a particular builder image is not compatible?
So as we can pretty print the black square, clearly explaining that a particular combination is deemed incompatible:

Note: It seems that UBI10 built image can run on UBI9 at the moment, it's probably due to the compatible gcc toolchain. I haven't found any promises about it being that way in the future, so I excluded UBI9 anyway.

This test run was made with this very branch:
quarkusio/quarkus-images#313 (comment)

Karm · 2025-11-27T23:37:41Z

I noticed the same GDB failure on "jdk-24-ubi - 3.21.4 - ubuntu-22.04" as tracked here: graalvm/mandrel#919

Karm · 2025-11-28T10:32:33Z

Error: The action 'Linux test (amd64 and arm64)' has timed out after 130 minutes.

Re-running...

jerboaa · 2025-11-28T10:53:31Z

Mandrel Builder Image / jdk-21-ubi - 3.21.4 - ubuntu-22.04-arm (pull_request) fails with:

> com.oracle.svm.core.util.VMError$HostedError: Class monitor_field_offset.Main480 has an invalid monitor field offset. Most likely, its objects are larger than supported.

jerboaa

OK

Karm · 2025-11-28T11:05:52Z

Mandrel Builder Image / jdk-21-ubi - 3.21.4 - ubuntu-22.04-arm (pull_request) fails with:
> com.oracle.svm.core.util.VMError$HostedError: Class monitor_field_offset.Main480 has an invalid monitor field offset. Most likely, its objects are larger than supported.

I don't think it does. It's an expected message from the previous test, doesn't it? It fails on the JFR...?

jerboaa · 2025-11-28T12:10:30Z

Mandrel Builder Image / jdk-21-ubi - 3.21.4 - ubuntu-22.04-arm (pull_request) fails with:
> com.oracle.svm.core.util.VMError$HostedError: Class monitor_field_offset.Main480 has an invalid monitor field offset. Most likely, its objects are larger than supported.
I don't think it does. It's an expected message from the previous test, doesn't it? It fails on the JFR...?

You are right. Fails with:

 Error:  Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 361.145 s <<< FAILURE! - in org.graalvm.tests.integration.JFRTest
Error:  jfrPerfContainerTest{TestInfo}  Time elapsed: 235.667 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: Timeout 10s was reached. Empty webpage does not contain string: `hello' ==> expected: <true> but was: <false>
	at org.graalvm.tests.integration.JFRTest.runBenchmarkForApp(JFRTest.java:377)
	at org.graalvm.tests.integration.JFRTest.startComparisonForBenchmark(JFRTest.java:275)
	at org.graalvm.tests.integration.JFRTest.jfrPerfTestRun(JFRTest.java:232)
	at org.graalvm.tests.integration.JFRTest.jfrPerfContainerTest(JFRTest.java:178)

Karm · 2025-12-01T14:39:19Z

Cannot reproduce the JFR thing locally:

$ mvn clean verify -Dtest=JFRTest#jfrPerfContainerTest 
-Ptestsuite-builder-image -Dquarkus.version=3.21.4 
-Dquarkus.native.builder-image=quay.io/quarkus/ubi9-quarkus-mandrel-builder-image:jdk-24
-Dquarkus.native.container-runtime=podman  | tee log.log

I can see in the downloaded log archive that Quarkus does start. I suspect we have some docker update at our hands. Comparing GHA logs and local logs line by line, one obvious difference is that the runtime container on my laptop is started with sudo and with podman while the GHA one with docker and without sudo. The test is quite old, so the change is the env. Gonna try to fix it.

~~Comparing the logs, both Q start.~~

aarch64 and amd64 CentOS 9 podman, both fine locally.

I can see the error now. I've been browsing a wrong GHA tarball. It seems that some GHA aarch64 runs suffer from:

docker: Error response from daemon: Conflict. The container name "/my-jfr-performance-runner" 
is already in use by container "82e5c5703ec3d50afda7ec1af70b605879ef539979c1882fea223cb0d064400a".
You have to remove (or rename) that container to be able to reuse that name.

i.e. likely a race condition in the JFR perf test container cleanup code. Gonna check that...

Karm · 2025-12-02T11:51:52Z

These things...
There is a stray slash on GHA docker, i.e. "/my-jfr-performance-runner" vs. "my-jfr-performance-runner".

…tainer name /my-jfr-performance-runner is already in use by container.... You have to remove (or rename) that container to be able to reuse that name.

Karm · 2025-12-02T14:19:22Z

The JFR container name thing seems gone with

and there is a new failure:

 PerfCheckTest.compareNativeAndJVMSerialGCTime:657 build-and-run.log log
 should not contain error or warning lines that are not whitelisted.
 See /home/runner/work/mandrel-integration-tests/mandrel-integration-tests/ts/testsuite/target/archived-logs/org.graalvm.tests.integration.PerfCheckTest/compareNativeAndJVMSerialGCTime/build-and-run.log
and check these offending 1 lines: 

14:04:01,868 ERROR (vert.x-eventloop-thread-2) [i.h.c.ControllerVerticle] Uncaught error: java.lang.NullPointerException

The error comes from Hyperfoil though, occures during our Quarkus java run:


java -Xlog:gc -XX:+UseSerialGC -Xmx64m -jar target/quarkus-app/quarkus-run.jar
Command: java -Xlog:gc -XX:+UseSerialGC -Xmx64m -jar target/quarkus-app/quarkus-run.jar
[0.002s][info][gc] Using Serial
[0.458s][info][gc] GC(0) Pause Young (Allocation Failure) 18M->4M(61M) 4.403ms
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by org.jboss.threads.JBossExecutors (file:/home/runner/work/mandrel-integration-tests/mandrel-integration-tests/ts/apps/quarkus-full-microprofile/target/quarkus-app/lib/main/org.jboss.threads.jboss-threads-3.8.0.Final.jar)
WARNING: Please consider reporting this to the maintainers of class org.jboss.threads.JBossExecutors
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
[0.852s][info][gc] GC(1) Pause Young (Allocation Failure) 21M->6M(61M) 7.046ms
[1.063s][info][gc] GC(2) Pause Full (Metadata GC Threshold) 18M->7M(61M) 16.374ms
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2025-12-02 14:03:29,148 INFO  [io.quarkus] (main) quarkus-full-microprofile 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.21.4) started in 1.102s. Listening on: http://0.0.0.0:8080
2025-12-02 14:03:29,153 INFO  [io.quarkus] (main) Profile prod activated. 
2025-12-02 14:03:29,153 INFO  [io.quarkus] (main) Installed features: [cdi, opentelemetry, rest, rest-client, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, smallrye-jwt, smallrye-metrics, smallrye-openapi, vertx]
[1.270s][info][gc] GC(3) Pause Young (Allocation Failure) 24M->8M(61M) 2.352ms
Command: docker run --name hyperfoil-container --rm --network=host quay.io/karmkarm/hyperfoil:0.25.2 standalone
14:03:51,122 INFO  (main) [i.h.Hyperfoil] Java: Red Hat, Inc. OpenJDK 64-Bit Server VM 11.0.20 11.0.20+8-LTS (/usr/lib/jvm/java-11-openjdk-11.0.20.0.8-3.el8.x86_64), CWD /home/jboss
14:03:51,127 INFO  (main) [i.h.Hyperfoil] Hyperfoil: 0.25.2 (0382c3340baf8205c3ad1d93ec475a106f6f2820)
14:03:51,127 INFO  (main) [i.h.Hyperfoil]            DISTRIBUTION:  /deployment
14:03:51,129 INFO  (main) [i.h.Hyperfoil]            ROOT_DIR:      /tmp/hyperfoil
14:03:51,129 INFO  (main) [i.h.Hyperfoil]            BENCHMARK_DIR: /tmp/hyperfoil/benchmark
14:03:51,130 INFO  (main) [i.h.Hyperfoil]            RUN_DIR:       /tmp/hyperfoil/run
14:03:51,130 INFO  (main) [i.h.Hyperfoil]            HOOKS_DIR:     /tmp/hyperfoil/hooks
14:03:51,131 INFO  (main) [i.h.Hyperfoil] Starting non-clustered Vert.x...
14:03:51,332 INFO  (main) [i.h.Hyperfoil] Deploying ControllerVerticle...
14:03:51,345 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerVerticle] Starting in directory /tmp/hyperfoil...
14:03:51,519 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerServer] Hyperfoil controller listening on http://runnervmfxdz0:8090
14:03:51,520 INFO  (vert.x-eventloop-thread-1) [i.h.Hyperfoil] ControllerVerticle deployed.
14:03:55,640 INFO  (vert.x-worker-thread-1) [i.h.c.u.PersistenceUtil] Stored benchmark 'microprofile-hyperfoil' in /tmp/hyperfoil/benchmark/microprofile-hyperfoil.yaml
14:03:55,657 INFO  (vert.x-eventloop-thread-0) [i.h.c.u.PersistenceUtil] Stored benchmark 'microprofile-hyperfoil' in /tmp/hyperfoil/run/0000/microprofile-hyperfoil.yaml
14:03:56,666 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerVerticle] Starting benchmark microprofile-hyperfoil - run 0000
14:03:56,666 INFO  (vert.x-eventloop-thread-2) [i.h.c.AgentVerticle] 206e3bf5-61b5-44ff-a592-d1cb7e832e9f Got reply from controller.
14:03:56,667 INFO  (vert.x-eventloop-thread-2) [i.h.c.AgentVerticle] Initializing agent
14:03:56,687 INFO  (vert.x-eventloop-thread-2) [i.h.h.c.HttpClientPoolImpl] Allocating 200 connections (max 200, buffer 0) in 1 executors to http://localhost:8080
14:03:56,844 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerPhase] 0000 main changing status NOT_STARTED to STARTING
14:03:56,849 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerPhase] 0000 main changing status STARTING to RUNNING
[29.164s][info][gc] GC(4) Pause Young (Allocation Failure) 25M->10M(61M) 4.877ms
[29.337s][info][gc] GC(5) Pause Full (Metadata GC Threshold) 22M->11M(61M) 26.191ms
[30.043s][info][gc] GC(6) Pause Young (Allocation Failure) 28M->12M(61M) 1.756ms
[30.987s][info][gc] GC(7) Pause Young (Allocation Failure) 29M->12M(61M) 2.040ms
2025-12-02 14:03:59,888 WARNING [io.qua.ope.run.exp.otl.sen.VertxGrpcSender] (vert.x-eventloop-thread-0) Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: Connection refused: localhost/127.0.0.1:4317
[32.842s][info][gc] GC(8) Pause Young (Allocation Failure) 29M->12M(61M) 2.015ms
14:04:01,842 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerPhase] 0000 main changing status RUNNING to FINISHING
14:04:01,847 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerPhase] 0000 main changing status FINISHING to FINISHED
14:04:01,858 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerPhase] 0000 main changing status FINISHED to TERMINATED
14:04:01,859 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerVerticle] 0000 All phases are terminated.
14:04:01,860 INFO  (vert.x-eventloop-thread-2) [i.h.c.AgentVerticle] Received agent reset
14:04:01,875 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerVerticle] Run 0000: completing stats for phase main
14:04:01,868 ERROR (vert.x-eventloop-thread-2) [i.h.c.ControllerVerticle] Uncaught error: java.lang.NullPointerException
    at io.hyperfoil.clustering.AgentVerticle.lambda$initBenchmark$9(AgentVerticle.java:247)
    at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
    at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
    at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43)
    at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)

14:04:01,922 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerVerticle] Run 0000 completed
14:04:02,082 INFO  (vert.x-eventloop-thread-0) [i.h.c.ControllerVerticle] Successfully persisted run 0000
2025-12-02 14:04:04,841 WARNING [io.qua.ope.run.exp.otl.sen.VertxGrpcSender] (vert.x-eventloop-thread-0) Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: Connection refused: localhost/127.0.0.1:4317
2025-12-02 14:04:04,849 INFO  [io.quarkus] (Shutdown thread) quarkus-full-microprofile stopped in 0.890s

Not gonna do anything about it in this PR.

Karm · 2025-12-02T15:11:29Z

So, the only failure is the gcc one now graalvm/mandrel#919

Merging.

Karm requested a review from jerboaa November 27, 2025 23:16

Karm mentioned this pull request Nov 27, 2025

Prepare the arrival of UBI 10 and do a lot of Spring cleaning #384

Closed

UBI10 runtime and builder

e1caeb2

Karm force-pushed the karm-prepare-ubi-10 branch from 4c712fc to e1caeb2 Compare November 27, 2025 23:36

jerboaa approved these changes Nov 28, 2025

View reviewed changes

Attempts to fix docker: Error response from daemon: Conflict. The con…

dd23cca

…tainer name /my-jfr-performance-runner is already in use by container.... You have to remove (or rename) that container to be able to reuse that name.

Karm merged commit 761c7d8 into master Dec 2, 2025
65 of 67 checks passed

Karm mentioned this pull request Dec 2, 2025

UBI10: Eventually enable it again #336

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

UBI10 runtime and builder #386

UBI10 runtime and builder #386

Uh oh!

Karm commented Nov 27, 2025 •

edited

Loading

Uh oh!

Karm commented Nov 27, 2025

Uh oh!

Karm commented Nov 28, 2025

Uh oh!

jerboaa commented Nov 28, 2025

Uh oh!

jerboaa left a comment

Uh oh!

Karm commented Nov 28, 2025

Uh oh!

jerboaa commented Nov 28, 2025

Uh oh!

Karm commented Dec 1, 2025 •

edited

Loading

Uh oh!

Karm commented Dec 2, 2025

Uh oh!

Karm commented Dec 2, 2025

Uh oh!

Karm commented Dec 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

UBI10 runtime and builder #386

UBI10 runtime and builder #386

Uh oh!

Conversation

Karm commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Karm commented Nov 27, 2025

Uh oh!

Karm commented Nov 28, 2025

Uh oh!

jerboaa commented Nov 28, 2025

Uh oh!

jerboaa left a comment

Choose a reason for hiding this comment

Uh oh!

Karm commented Nov 28, 2025

Uh oh!

jerboaa commented Nov 28, 2025

Uh oh!

Karm commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Karm commented Dec 2, 2025

Uh oh!

Karm commented Dec 2, 2025

Uh oh!

Karm commented Dec 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Karm commented Nov 27, 2025 •

edited

Loading

Karm commented Dec 1, 2025 •

edited

Loading