@EmbeddedKafka + Kafka 3.6.0 Compatibility: Seeking Advice on NoClassDefFoundError

[HenriqueKazzak]

Hello,
I'm encountering a NoClassDefFoundError when trying to use @EmbeddedKafka with Spring Boot 2.7.18 and Java 11. I'm using kafka-clients:3.6.0. This specific version of kafka-clients is mandated by company policy, so I can't easily change it.
I haven't been able to find clear documentation regarding the compatibility of this specific kafka-clients version with Spring Boot 2.7.18 and Java 11, especially in the context of using @EmbeddedKafka for testing.
Dependency snippets from my project:

// My pom
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>

// From company kafka lib
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
        </exclusion>
    </exclusions>
</dependency>

// From company pom dependencies (properties)
<properties>
    <spring-kafka.version>2.9.13</spring-kafka.version>
    <confluent.version>7.7.1</confluent.version>
    <avro.version>1.11.4</avro.version>
    <kafka-clients.version>3.6.0</kafka-clients.version>
</properties>

// From company pom dependencies
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>${spring-kafka.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>${kafka-clients.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>${kafka-clients.version}</version>
</dependency>

When I run my tests, I get the following error:

Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/common/requests/AlterIsrRequest$Builder
    at kafka.server.KafkaServer.startup(KafkaServer.scala:313)
    at kafka.utils.TestUtils$.createServer(TestUtils.scala:177)
    at kafka.utils.TestUtils$.createServer(TestUtils.scala:172)
    at kafka.utils.TestUtils.createServer(TestUtils.scala)
    at org.springframework.kafka.test.EmbeddedKafkaBroker.afterPropertiesSet(EmbeddedKafkaBroker.java:367)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
    ... 80 more
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.requests.AlterIsrRequest$Builder
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    ... 87 more

Here's a snippet from my dependency tree (focused on spring-kafka-test):

[INFO] +- org.springframework.kafka:spring-kafka-test:jar:2.9.13:test
[INFO] |  +- org.springframework:spring-context:jar:5.3.31:compile
[INFO] |  |  \- org.springframework:spring-expression:jar:5.3.31:compile
[INFO] |  +- org.springframework:spring-test:jar:5.3.31:test
[INFO] |  +- org.springframework.retry:spring-retry:jar:1.3.1:compile
[INFO] |  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.6.3:test
[INFO] |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.6.3:test
[INFO] |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:test
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.119.Final:compile
[INFO] |  |  |  +- io.netty:netty-common:jar:4.1.119.Final:compile
[INFO] |  |  |  +- io.netty:netty-resolver:jar:4.1.101.Final:compile
[INFO] |  |  |  +- io.netty:netty-buffer:jar:4.1.101.Final:compile
[INFO] |  |  |  +- io.netty:netty-transport:jar:4.1.101.Final:compile
[INFO] |  |  |  +- io.netty:netty-transport-native-unix-common:jar:4.1.101.Final:compile
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.101.Final:compile
[INFO] |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.101.Final:test
[INFO] |  |     \- io.netty:netty-transport-classes-epoll:jar:4.1.101.Final:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:test:3.1.2:test
[INFO] |  +- org.apache.kafka:kafka-metadata:jar:3.1.2:test
[INFO] |  |  +- org.apache.kafka:kafka-server-common:jar:3.1.2:test
[INFO] |  |  +- org.apache.kafka:kafka-raft:jar:3.1.2:test
[INFO] |  |  \- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] |  +- org.apache.kafka:kafka-streams-test-utils:jar:3.1.2:test
[INFO] |  |  \- org.apache.kafka:kafka-streams:jar:3.6.0:test
[INFO] |  |     \- org.rocksdb:rocksdbjni:jar:7.9.2:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:3.1.2:test
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.13.6:test
[INFO] |  |  +- org.apache.kafka:kafka-storage:jar:3.1.2:test
[INFO] |  |  |  \- org.apache.kafka:kafka-storage-api:jar:3.1.2:test
[INFO] |  |  +- net.sourceforge.argparse4j:argparse4j:jar:0.7.0:test
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.13.5:test
[INFO] |  |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO] |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.13.5:compile
[INFO] |  |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:compile
[INFO] |  |  +- org.bitbucket.b_c:jose4j:jar:0.7.8:test
[INFO] |  |  +- org.scala-lang.modules:scala-collection-compat_2.13:jar:2.4.4:test
[INFO] |  |  +- org.scala-lang.modules:scala-java8-compat_2.13:jar:1.0.0:test
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.13.6:test
[INFO] |  |  +- com.typesafe.scala-logging:scala-logging_2.13:jar:3.9.3:test
[INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:4.2.22:test
[INFO] |  |  \- commons-cli:commons-cli:jar:1.4:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:test:3.1.2:test
[INFO] |  \- com.google.code.findbugs:jsr305:jar:3.0.2:compile

As you can see from the dependency tree, spring-kafka-test 2.9.13 pulls in kafka-clients:3.1.2 for testing purposes. If I explicitly set kafka-clients to version 3.1.2, the tests work fine.

My core question is:
Is it possible to use spring-kafka and spring-kafka-test 2.9.13 with Spring Boot 2.7.18, Java 11, and kafka-clients 3.6.0? Or do I need to use a different version combination to ensure compatibility with @EmbeddedKafka?
Any insights or recommendations would be greatly appreciated. Thanks!

[artembilan]

Try to follow these recommendations: https://docs.spring.io/spring-kafka/docs/2.9.13/reference/html/#update-deps.
But from big picture, that Spring Boot version, including all of its Spring dependencies, is out of Open Source Support, so we cannot help you from here.
Unless you can use dedicated commercial channels support: https://enterprise.spring.io/

[HenriqueKazzak]

I added this dependency, just like the documentation, but it didn't work, the not founded class is in the jar of kafka-clients

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-server-common</artifactId>
            <classifier>test</classifier>
            <scope>test</scope>
            <version>3.6.0</version>
        </dependency>

[HenriqueKazzak]

My bad, it didn't work. I was importing kafka-clients:3.1.2 and didn't notice it. Maybe it isn't compatible.

[sobychacko]

It would be much better if you could update to a supported version, start from that baseline, and see if it works.